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1  SUMMARY 

It  Is  possible  for  a  relatively  small  cost  In  personnel  time  to  ob¬ 
tain  data  for  software  engineering  and  computer  science  research  as  a  by¬ 
product  of  existing  USACSC  reporting  practices.  These  data  when  manipulated 
by  automated  systems  which  already  exist  can  provide  many  of  the  data 
elements  describing  the  computer  systems  built  at  the  Command,  the  resources 
required  to  complete  them,  and  the  development  and  maintenance  environment. 
These  three  aspects  of  the  software  development  process  are  the  principal 
components  of  any  research  data  structure. 

Software  product  data,  which  include  measures  of  size,  type,  and 
complexity  are  best  obtained  from  the  programs  themselves.  This  can  be 
accomplished  by  making  copies  of  released  systems.  Reliability  data  can  be 
obtained  from  a  modified  Incident  Report.  In  both  cases,  however,  and 
to  obtain  data  describing  the  system  documentation,  it  will  be  necessary 
to  use  supplementary  data  collection  instruments. 

ft 

Personnel  hours  expended  in  system  development  and  maintenance  are 
best  obtained  by  implementing  a  standard  Job  Number  format  in  the  Resource 
Management  Accounting,  Reporting  and  Control  System  (REMARCS).  This  will 
allow  capturing  the  hours  spent  In  various  life  cycle  phases  at  the  program 
and  System  Change  Request  levels  of  detail.  Supplementary  information 
will  be  required  to  identify  personnel  experience. 

Computer  resources  used  are  presently  recorded  by  the  operating  system 
accounting  subsystem.  In  this  case  and  in  the  case  of  the  REMARCS  data 
some  new  software  will  be  needed  to  trasform  It  from  its  recorded  format 
to  one  more  useful  for  research  needs.  A  data  retrieval  system  is  avail¬ 
able  once  the  data  have  been  extracted. 

Environment  Data  should  be  collected  when  dictated  by  changes  In 
the  operating  environment.  Specially  designed  forms  snould  be  used  for 
this  purpose. 
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2  INTRODUCTION 

Several  researchers  have  observed  that  the  availability  of  good 
quality  historical  data  Is  of  prime  Importance  to  conducting  research  in 
software  phenomena  [1,  2,  3].  It  has  also  been  noted  that  this  vital  com¬ 
modity  is  in  extremely  short  supply  [4,  5].  In  a  previous  study  [6]  the 
USACSC,  AIRMICS  surveyed  the  availability  of  software  data  that  might  be 
used  in  its  research  programs.  The  study  results  indicated  that  while  some 
data  exist  they  were  not  believed  to  satisfy  the  needs  of  the  comprehensive 
research  program  at  AIRMICS.  The  quality  of  the  data  was  adversely  affected 
by  definitions  that  were  difficult  to  apply  at  USACSC,  poorly  controlled 
collection  practices,  lack  of  precision  and  incompatibility  with  the  appli¬ 
cations  and  development  environment  existant  at  the  Command.  It  was  be¬ 
lieved  that  the  state  of  the  art  In  software  science  and  engineering  does 
not  permit  the  correction  of  data  to  make  it  usable  at  AIRMICS. 

After  concluding  that  data  suitable  for  research  do  not  exist  outside 
of  the  Command,  attention  was  directed  to  the  data  available  within  the 
Command.  The  objective  of  this  study,  then,  is  to  investigate  the  record 
keeping  practices  that  are  already  an  integral  part  of  the  operation  of 
USACSC  to  learn  if  data  that  can  be  used  in  research  are  generated  during 
the  normal  conduct  of  business.  Any  organization  records  certain  aspects 
of  its  operations  in  order  to  manage  effectively.  Since  many  items  that 
are  part  of  the  management  domain  are  also  needed  for  research,  it  is 
expected  that  valuable  data  can  be  acquired  with  little  or  no  additional 
effort. 

The  Identification  of  potential  research  data  was  accomplished  by 
first  analyzing  existing  AIRMICS  research  programs  to  learn  what  data  were 
needed.  Then  the  data  obtained  as  part  of  the  management  process  were 
Identified.  Comparing  the  two  sets,  it  was  possible  to  propose  ways  to 
utilize  the  management  data  to  satisfy  the  needs  of  the  research  program. 
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Each  area  of  research  was  analyzed  for  data  usage.  The  data  were 
separated  Into  broad  classifications: 

Product 

Resources 

Environment 

These  classifications  are  convenient  because  they  are  relatively 
easy  to  identify  but  also  because  they  tend  to  be  recorded  in  distinct 
parts  of  the  management  process.  For  example,  resources  which  include 
items  such  as  project  hours  spent  by  job  category  are  the  domain  of  the 
Comptroller;  whereas,  product,  which  is  the  system  of  programs,  is 
located  at  the  system  developers  and  is  directly  measurable  when  it  is 
released  to  the  field. 

The  broad  classifications  were  systematically  broken  down  into 
groups  until  individual  items  were  identified.  The  data  structure  is  de¬ 
signed  so  that  the  items  can  be  combined  to  form  values  that  are  meaningful 
to  different  research  topics.  Sometimes  this  is  difficult  because  it 
produces  a  conflict  with  the  software  construction  process.  For  example, 
to  study  the  process  by  which  errors  are  introduced  to  a  software  product, 
it  may  be  necessary  to  trace  a  problem  in  a  released  version  of  a  system 
back  to  a  System  Change  Request  or  to  a  user's  requirements.  This  is  a 
natural  sequence  in  the  geneology  of  software  problems.  However,  require¬ 
ments  and  change  requests  tend  to  be  functional  from  the  system  point  of 
v.iew.  Often  specific  functions  are  implemented  using  more  than  one  program. 
When  a  new  system  is  being  developed  or  when  several  changes  are  being 
installed  at  the  same  time,  it  becomes  difficult  if  not  impossible  to  trace 
problems  back  to  requirements.  The  data  classification  scheme  provides  a 
framework  for  reconciling  the  need  for  data  with  the  way  in  which  it  is 
generated  (or  the  way  In  which  the  work  Is  carried  out). 
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THE  NEED  FOR  DATA 

The  data  that  are  necessary  for  investigating  the  software  development 
process  can  be  divided  into  three  categories: 

1.  Metrics  describing  the  net  product  of  the  development  effort. 

These  are  the  attributes  of  the  software  system: 

-  size 

-  reliability 

-  complexity 

-  transportability 

-  etc. 

2.  Measures  of  the  resources  required  to  develop  and  maintain  the 
systems : 

-  personnel  hours 

-  computer  time 

-  support  software 

-  etc. 

3.  Descriptions  of  the  development  environment: 

-  development  standards  including  documentation  and  testing 

-  modern  programing  practices 

-  accessabillty  to  the  computer 

-  etc. 

It  is  not  possible  to  conduct  successful  research  projects  with 
fragmentary  descriptions  drawn  from  only  one  or  two  of  the  above  cate¬ 
gories.  Whereas  we  can  satisfy  some  immediate  or  limited  objectives  that 
way,  we  cannot  address  the  underlying  purpose  of  the  AIRMICS  mission: 

Improve  the  quality  of  software  In  the  field  and  research  more  efficient 
methods  for  designing,  building,  and  managing  the  development  of  systems. 

We  have  selected  specific  research  programs  now  being  conducted  at  AIRMICS  . 
as  a  means  for  focusing  our  attention  on  data  needs,  we  will,  nevertheless, 
be  considering  the  larger  objectives  of  the  Institute. 
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The  immediate  objectives  for  this  task  center  around  two  major  areas 
of  AIRMICS  research  that  will  benefit  from  the  availability  of  USACSC 
data.  These  are  studies  involving  Mutation  Analysis  and  Software  Science. 
Studies  related  to  quality  metrics  and  life  cycle  management  will  also 
benefit  from  addtional  data. 

3.1  MUTATION  ANALYSIS 

This  area  of  research  is  Intended  to  increase  system  reliability 
by  improving  the  quality  of  test  data.  Input  data  sets  are  executed  by 
programs  with  systematically  Introduced  changes  or  mutations.  Test  data 
that  can  detect  a  complete  sequence  of  program  mutations  is  hypothesized 
to  be  capable  of  detecting  any  errors  likely  to  exist  in  a  software  system 
before  the  system  is  released.  Present  attention  is  directed  toward 
proving  this  hypothesis  on  the  basis  of  individual  programs.  Having  accom¬ 
plished  that,  the  concept  will  be  demonstrated  with  complete  systems. 

In  order  to  improve  the  selection  of  program  mutation  operators,  it 
will  be  necessary  to  obtain  data  that  will  allow  Investigators  to  demon¬ 
strate  that  the  classes  of  errors  addressed  by  the  mutation  operators  are 
those  actually  made  by  programmers  at  USACSC.  This  will  be  accomplished 
by  documenting  the  process  through  which  problems  discovered  in  operational 
programs  are  corrected  by  appropriate  changes  in  the  program  code.  By 
analyzing  such  data  it  should  be  possible  using  Mutation  Analysis  to 
essentially  reverse  the  process.  Mutational  operators  will  be  created 
that  change  programs  during  the  testing  process  in  a  way  that  will  produce 
errors  that  are  representative  of  the  reported  problems. 

We  would  like  to  obtain  error  data  that  will  accomplish  two  purposes: 

•  Describe  the  errors  that  occur  in  operation  programs  in  a  way 
that  Is  consistent  with  the  classes  of  errors  addressed  by  the 
mutation  operators. 

•  Describe  the  relationship  between  errors  and  code  changes  to 
determine  if  the  use  of  simple  mutations  are  adequate  to  repro¬ 
duce  all  practical  error  types. 
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Appendix  I  describes  the  classes  of  errors  for  which  mutation 
operators  have  been  designed.  These  errors  are  considered  to  be  repre¬ 
sentative  and  the  associated  mutation  operators  are  believed  to  detect 
them  when  properly  Interpreted.  That  is,  the  mutations  produce  results 
which  will  either  be  detected  by  a  chosen  test  data  set  or  will  be  flagged 
for  further  analysis. 

Appendix  II  shows  a  way  of  associating  measurable  error  attributes 
with  the  error  classes.  These  attributes  establish  the  need  for  error 
data  presented  by  Mutation  Analysis. 

3.2  SOFTWARE  SCIENCE 

AIRMICS  projects  are  testing  several  applications  of  the  principals 
of  Software  Science.  These  include  the  investigation  of  relationships 
between  the  various  Software  Science  metrics  and  such  things  as  program¬ 
ming  effort,  complexity,  reliability,  and  clarity. 

The  Software  Science  metrics  are  derived  from  the  analysis  of  the 
source  programs.  Therefore  obtaining  this  data  becomes  a  matter  of  ac¬ 
quiring  copies  of  the  programs  and  processing  them  with  a  special  analyzer 
program.  Activities  are  already  under  way  to  obtain  both  the  analyzer 
and  the  source  programs.  Our  efforts,  therefore,  will  be  directed  toward 
obtaining  data  to  be  correlated  with  the  Software  Science  measures. 

An  important  distinction  between  the  data  needed  for  the  Mutation 
Analysis  and  that  associated  with  the  Software  Science  studies  is  that 
the  former  are  derived  from  operational  programs  while  the  latter  include 
information  describing  the  program  development.  Although  it  is  expected 
that  Mutation  Analysis  will  be  applied  in  the  development  of  programs,  the 
development  phase  is  not  considered  the  appropriate  time  to  obtain  error 
data.  This  is  because  the  programs  are  often  in  a  state  of  change  and 
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the  number  of  trivial  errors  or  bugs  detected  in  this  phase  of  the  life 
cycle  is  not  considered  to  be  representati ve  of  the  reliability  of  the 
system.  Such  a  relationship  might  occur  during  the  final  stages  of  test¬ 
ing,  but  for  now  we  will  recommend  that  error  collection  be  initiated 
after  the  system  is  released. 

In  both  cases  an  important  part  of  the  data  collection  process  is 
correlating  the  data  obtained  from  different  sources.  Care  must  be  taken 
to  ensure  that  the  errors,  effort,  and  other  measures  are  associated  with 
a  specific  program  version.  There  is  danger  that  with  programs  undergoing 
constant  correction  and  modification,  the  data  describing  the  resources, 
environment,  and  errors  will  not  be  associated  with  the  proper  program 
versions.  This  could  significantly  affect  the  results  of  any  investiga¬ 
tions  using  such  data. 

Appendix  III  describes  data  needs  associated  with  the  Software 
Science  and  other  AIRMICS  programs. 
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THE  AVAILABILITY  OF  DATA 

The  U.S.  Army  Computer  Systems  Command  has  the  responsibility  for 
developing  and  maintaining  Standard  Army  Multi -Command  Management  Informa¬ 
tion  Systems  (STAMMIS).  Standard  systems  operate  across  major  commands 
and  support  primary  Army  missions.  Typical  systems  are  the  Standard 
Integrated  Personnel  System  (SIDPERS)  and  the  Standard  Army  Intermediate 
Level  Supply  System  (SAILS).  The  functions  supported  by  the  computer 
systems  (personnel  and  logistics)  are  the  responsibilities  of  specific 
Army  organizations,  but  the  day-to-day  execution  of  the  functions  occurs 
at  Army  installations  all  over  the  world.  This  gives  rise  to  a  matrix 
type  of  organizational  responsibility  where  specific  supporting  functions 
are  the  responsibility  of  one  organization  and  the  line  or  supported  func¬ 
tion  is  the  responsibility  of  another.  In  the  examples  cited,  the  functions 
of  personnel  administration  and  supply  support  the  operations  of  all 
Army  installations. 

In  keeping  with  the  matrix  concept  of  functional  responsibilities, 
the  functional  specification  of  each  STAMMIS  is  assigned  to  an  organi¬ 
zation  that  has  jurisdiction  over  the  STAMMIS  mission.  The  functional 
organization  is  designated  the  Proponent  Agency  (PA).  It  is  responsible 
for  the  proper  interpretation  of  all  system  requirements  and  has  final 
say  on  the  system  functional  specifications.  The  actual  user  of  the  system 
is  the  field  installation.  As  in  the  case  of  any  user  or  supported  organi¬ 
zation,  It  will  have  a  vested  interest  in  the  functioning  of  any  system 
because  to  an  ever  increasing  extent  the  computer  support  system  has  an 
impact  on  how  well  the  organization  can  complete  its  mission. 

The  development  and  maintenance  of  the  computer  system  itself  is  the 
responsibility  of  the  USACSC.  The  -.Command  is  organized  in  part  along 
functional  lines.  Each  of  the  primary  Army  functions  supported  by  standard 
automated  systems  are  assigned  to  specific  elements  within  the  Command. 
Other  USACSC  elements  are  responsible  for  executive  software  develop¬ 
ment,  computer  system  performance  analysis,  quality  assurance,  configur¬ 
ation  management  and  customer  assistance. 
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Each  computer  system  has  a  Proponent  Agency  for  functional  specifi¬ 
cations  and  an  Assigned  System  Developer  (ASD).  The  ASD  is  responsible 
for  building  and  maintaining  a  computer  system  that  complies  with  the  PA's 
functional  specifications.  The  PA  Is  responsible  for  not  only  the  initial 
version  of  the  system,  but  all  subsequent  functional  changes  as  well. 

Errors  In  the  system  or  execution  failures  can  be  temporarily  devastating 
to  a  line  organization.  As  would  be  expected,  the  user  keeps  a  constant 
pressure  on  the  ASD  to  maximize  the  reliability  of  operational  systems. 
Because  of  his  everyday  involvement  with  a  computer  system,  a  user  is 
sensitive  to  how  the  system  functions.  Many  suggestions  for  Improvements 
originate  with  the  user. 

However,  the  Individual  users  are  often  guided  by  local  considera¬ 
tions  that  may  not  be  generally  applicable  and  that  if  Implemented  may 
reduce  the  performance  of  the  system  to  a  level  unacceptable  to  other 
users.  The  responsibility  for  making  these  judgments  remains  with  the  PA. 
The  degree  to  which  a  STAMMIS  satisfies  all  system  users  by  providing 
useful,  timely  information  with  high  reliability  U  the  measure  of  how 
successfully  the  Proponent  Agency,  Assigned  System  Developer,  and  users 
have  worked  together  to  satisfy  often  conflicting  requirements. 

4.1  THE  SYSTEM  DEVELOPMENT  AND  MAINTENANCE  PROCESS 

The  software  system,  just  as  any  other  Item  purchased  or  developed 
by  the  government,  Is  considered  to  have  a  finite  useful  life.  The  soft¬ 
ware  life  cycle  begins  when  a  need  is  recognized  and  it  ends  when  the 
system  designed  to  meet  the  need  is  no  longer  useful. 

A  number  of  minor  cycles  or  phases  have  been  Identified  within  the 
system  life  span.  These  life  cycle  phases  define  the  need  to  be  satisfied, 
investigate  alternative  methods  for  satisfying  the  need,  produce  a  specific 
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design.  Implement  the  design,  and  support  the  system  after  It  becomes 
operational . 

The  USACSC  defines  the  life  cycle  phases  of  computer  systems  In 
accordance  with  DoD  Directive  7920.1  (Life  Cycle  Management  of  Automated 
Information  Systems  dated  17  October  1978)  [7],  The  following  phases  are 
identified: 

Mission  Analysis/ Project  Initiation 
Concept  Development 
Definition/Design 
System  Development 
Deployment  and  Operation 

For  the  purpose  of  data  analysis  we  will  divide  the  system  life 
cycle  into  two  parts:  Development  and  Maintenance. 

The  Development  Cycle  of  a  software  system  includes  the  first  four 
phases  described  above.  It  ends  when  the  system  is  declared  ready  for 
operational  use. 

Although  the  Development  Phase  has  received  the  greater  amount  of 
attention  in  the  software  engineering  literature,  it  comprises  only  about 
30  percent  of  the  average  total  life  cycle  time  span  and  consumes  only 
40  percent  of  the  total  resources.  It  is  important  to  recognize  this 
because  it  directs  our  attention  to  the  need  for  documenting  and  controlling 
the  maintenance  part  of  the  system  life  cycle  as  well  as  the  development 
part. 


The  Maintenance  Phase  extends  until  the  system  Is  scrapped.  The 
average  system  life  is  on  the  order  of  eight  years*.  During  that  time 
the  system  usually  undergoes  significant  change  to  accomodate  changes  In 
regulations,  procedures,  operating  equipment  and  software  and  to  Include 


*[7]  p.3-33 


new  capabilities.  Sometimes  a  system  becomes  quite  different  from 
its  original  version  through  this  evolutionary  process.  For  this  reason 
there  is  some  disagreement  among  software  analysts  as  to  what  exactly 
constitutes  the  life  cycle  of  a  system.  A  system  usually  retains  its 
identity  as  long  as  It  can  be  changed  in  part  to  accomodate  a  new  need. 

It  does  not  matter  how  many  previous  changes  have  occurred.  It  is  only  when 
the  size  of  a  given  change  is  such  that  in  addition  to  other  considerations 
it  becomes  preferable  to  rebuild  the  system  rather  than  modify  It.  Some 
of  the  other  considerations  that  would  lead  to  a  new  system  Include  a 
tendency  for  a  system  to  grow  In  size  and  decrease  in  performance  that 
occurs  after  many  modifications.  This  tendency  is  usually  associated 
with  a  decrease  in  reliability  and  an  increase  in  complexity. 

The  Maintenance  Cycle,  then,  includes  two  aspects:  the  correction 
of  errors  that  are  found  in  the  operational  system,  and  the  modification 
of  the  system  to  meet  new  requirements.  Together,  these  activities 
require  about  60  percent  of  the  total  life  cycle  effort. 

4.1.1  The  Development  Cycle 

The  life  cycle  phases  comprising  the  Development  Cycle,  as  defined 
in  DoD  Directive  7920.1  Include: 

Mission  Analysis/Project  Initiation 
Concept  Development 
Definition/ Design 
System  Development 

Table  1  describes  these  phases  in  greater  detail. 

Figure  1  shows  how  the  effort  is  distributed  over  time  and  uses 
notation  that  Is  presently  In  use  at  USACSC. 

As  was  mentioned  In  the  preceding  section,  the  programs  of  primary 
subjects  of  AIRMICS  research  are  the  large  Standard  Army  Multi -Command 
Management  Information  Systems.  These  are  associated  with  large 
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TABLE  1 


DEFINITIONS  OF  DEVELOPMENT  CYCLE  PHASES* 


Mission  Analysis/Project  Initiation 

The  purpose  of  this  phase  is  to  identify  a  mission  element 
need  (set  of  functional  requirements);  validate  that  need; 
and  recommend  the  exploration  of  alternative  functional 
concepts  to  satisfy  the  need.  This  phase  is  completed 
upon  approval  of  the  Mission  Element  Need  Statement  (MENS) 
at  Milestone  0  at  a  prescribed  organizational  level  and 
issuance  of  authority  to  explore  and  develop  alternative 
concepts. 

Concept  Development 

The  purpose  of  this  phase  is  to  synthesize  (or  solicit) 
and  evaluate  alternative  methods  to  accomplish  the 
function  shown  in  the  approved  MENS  and  to  recormiend 
one  (or  more)  feasible  concepts  for  further  exploration. 

A  determination  is  made  whether  several  alternative 
concepts  should  be  demonstrated  or  that  demonstration 
should  be  omitted. 

Definition/ Design 

The  purpose  of  this  phase  is  to  define  fully  the 
functional  requirements  (system/subsystem)  speci¬ 
fications)  and  to  design  an  operable  Automated 
Information  System  (AIS).  This  phase  is  completed 
when  ADP  and  telecommunications  technical  adequacy 
has  been  validated  and  upon  issuance  of  approval 
at  Milestone  II  at  a  prescribed  organizational  level 
to  develop  fully  the  system. 

System  Development 

The  purpose  of  this  phase  Is  to  develop,  integrate, 
test  and  evaluate  the  ADP  system  and  the  total  AIS. 

This  phase  is  completed  upon  approval  of  the  AIS  by 
appropriate  functional  officials  as  satisfying  the 
mission  need;  and  Issuance  of  approval  at  Milestone 
III  at  an  appropriate  organizational  level  to  deploy 
and  operate  the  approved  AIS. 


Source:  [7]  p.1-4 


Figure  1  Time  Distribution  of  Life  Cycle  Phases* 


Source:  [7]  pp  3-3.1  and  3-34.1 
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development  efforts  and  therefore  the  development  process  has  been  very 
carefully  defined.  Associated  with  the  activities  are  milestones  and 
reviews  that  are  necessary  to  ensure  that  resources  are  being  spent 
effectively.  Since  the  activities,  while  many  in  number,  are  identifiable 
with  a  specific  project,  the  description  of  the  data  requirements  are 
relatively  straightforward.  Of  course  the  ability  to  define  data  needs 
does  not  guarantee  obtaining  them,  but  the  development  process  is  in 
general  easier  to  define  and  track  than  the  Maintenance  Cycle. 

4.1.2  The  Maintenance  Cycle 

The  last  system  life  cycle  phase  is  Deployment  and  Operation, 
which,  according  to  DoD  Directive  7920.1,  includes: 

#  Implementation  of  the  approved  operational  plan, 
including  extension/installation  at  other  sites 

#  Continuation  of  approved  operations 

#  Adequate  budgeting 

e  Controlling  all  changes  and  maintaining  and  modifying  the 
Army  information  system  during  its  remaining  life  using 
well  defined  configuration  management  procedures 

The  objective  of  the  activities  comprising  the  Maintenance  Cycle  is  to 
make  the  system  as  responsive  as  possible  to  the  user's  requirements. 
Maintenance  activities  include  correction  of  errors  in  the  system,  modifying 
the  system  to  make  it  more  efficient  and  to  reflect  changes  in  procedures, 
and  providing  the  user  with  any  assistance  needed  in  the  operation  of  the 
system. 

4. 1.2.1  System  Problems 

To  the  user  the  first  test  of  a  computer  system  is  that  it  executes 
to  completion  using  his  operational  data.  On  successful  completion  of  his 
workload,  the  user  can  then  consider  the  other  qualities  of  the  system  such 
as  the  utility  of  the  information  and  its  format  and  the  speed  and  efficiency 
of  the  system.  He  may,  consequently,  offer  suggestions  for  improving  these 
measures.  However,  if  the  system  is  unreliable,  it  ceases  to  be  an  asset 
to  his  organization  and,  indeed,  it  can  quickly  become  an  enormous  burden. 
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There  are  countless  reasons  that  a  system  does  not  execute 
properly.  Sometimes  In  spite  of  training  and  documentation  the  user 
does  not  perform  his  duties  correctly.  Other  times  there  are  diffi¬ 
culties  with  the  computer  hardware.  And,  of  course,  there  are  times 
when  some  aspect  of  the  system  from  specifications  to  computer  programs 
is  at  fault. 

When  a  problem  occurs,  the  first  objective  is  to  get  the  system 
functioning  as  quickly  as  possible.  At  the  USACSC  the  organization  with 
responsibility  for  doing  this  is  the  Customer  Assistance  Office.  This 
organization  is  the  first  and  only  authorized  point  of  contact  for  any 
data  processing  installation  experiencing  difficulty  with  a  standard 
system.  When  a  problem  is  reported,  the  CAO  first  determines  If  the 
problem  can  be  resolved  quickly*,  if  not,  the  office  gets  highest  priority 
to  obtain  whatever  assistance  is  needed  from  any  organization  at  the 
USACSC  to  get  the  installation  running  as  quickly  as  possible. 

In  order  to  do  its  job  effectively,  the  CAO  has  developed 
procedures  based  on  the  Incident  Report.  Each  reported  incident  Is 
recorded  and  classified.  If  the  problem  is  clearable  on  the  telephone, 
it  is  so  indicated  and  the  incident  is  closed.  If  more  action  is 
required,  this  is  undertaken  and  a  series  of  tracking  and  reporting 
activities  is  executed  that  is  designed  to  ensure  that  all  reported 
Incidents  are  properly  resolved.  The  primary  purpose  of  this  process 
is  to  ensure  responsiveness  to  the  needs  of  the  data  processing  In- 
installations  when  problems  occur  while  processing  standard  Army  systems. 


*  There  is  a  checklist  of  possible  reasons  including  failure  to 

Incorporate  system  changes,  failure  to  follow  certain  procedures  or 
even  temporary  fixes  or  ways  around  the  problem  discovered  by 
working  with  other  sites. 


4-8 


The  Customer  Assistance  Office  may  help  the  user  by  reviewing 
information  available  to  the  user  or  by  providing  new  Information  not 
yet  generally  available.  As  will  be  shown  In  a  later  section,  there  are 
several  possible  outcomes  of  a  reported  problem. 

4. 1.2. 2  System  Changes 

Periodically  it  is  necessary  to  make  changes  to  an  operational  system. 
These  changes  are  described  in  System  Change  Requests  (SCRs)*  and  may  be 
motivated  by  any  of  the  following  conditions: 

•  Errors  in  the  released  system 

•  Changes  in  regulations 

•  Changes  to  the  computer  system  hardware  or  its  operating 
software 

•  The  desire  to  obtain  new  information 

•  The  desire  to  change  the  presentation  of  some  of  the  current 
Information 

•  The  desire  to  increase  the  speed  of  the  system  execution  or  to 
reduce  the  amount  of  memory  required  to  execute  the  system 

The  US ACS C  utilizes  a  quarterly  cycle  of  changes  to  operational 
systems.  Every  three  months  a  completely  new  system  is  released  to  the 
field.  Each  release  is  called  a  System  Change  Package  (SCP)  and  it 
incorporates  all  authorized  changes  to  all  systems  that  have  been 
approved  for  release  at  that  time. 

The  system  change  process  and  its  end  product,  the  SCP,  is 
responsible  for  much  of  the  data  that  Is  useful  for  research.  It  is  the 
principal  activity  of  maintenance  and  represents  the  largest  application 
of  resources.  Therefore,  It  is  necessary  to  understand  how  the  process 
is  accomplished  at  USACSC  and  to  know  what  data  are  presently  available 
to  describe  it. 


★ 

In  addition  to  SCRs  which  are  routine  functional  changes  the  USACSC  initiates 
changes  to  systems  using  Technical  Change  Requests  (TCRs)  and  Emergency/ 
Urgent  Change  Requests  (£/UCRs). 


The  Proponent  Agency  for  any  given  system  has  primary 
responsibility  for  the  functional  specifications  of  that  system. 

Therefore,  the  decision  to  incorporate  changes  into  a  system  and  to 
schedule  the  change  for  a  specific  SCP  is  made  by  the  Proponent  Agency. 

Figure  2  describes  the  change  process. 

It  is  important  to  the  understanding  of  the  data  collection 
problem  to  recognize  that  the  process  represented  by  Figure  2  is 
continuous.  SCRs  are  constantly  being  submitted  and  changes  are  always 
being  worked  on.  This  requires  careful  identification  of  work  and 
product  units  so  that  effort  can  be  properly  related  to  its  result. 

Versions  of  a  system  are  identified  by  change  packages  which  in 
turn  are  labeled  by  broadcast  date.  A  given  version  usually  includes 
a  collection  of  SCRs,  TCRs,  and  E/UCRs.  The  selection  of  these  for  in¬ 
clusion  in  a  change  package  depends  on  their  priority  and  the  time  and 
effort  required. 

The  time  and  effort  required  for  an  SCR  is  initially  estimated 
using  a  procedure  that  is  part  of  the  SCR  form.  This  gives  the  SCR 
Change  Review  Board  some  indication  of  the  effort  required  for  any  given 
change.  These  initial  estimates  are  later  reviewed  by  the  Assigned  System 
Developer  (ASD)  in  preparation  for  inclusion  in  an  SCP. 

The  SCR  Change  Review  Board  assigns  priorities  to  all  routine 
SCRs.  These  are  used  by  the  ASD  in  scheduling  the  changes. 

In  addition  to  routine  changes,  change  packages  include  technical 
changes  (needed  to  accomodate  changes  in  equipment  or  operating  software 
or  to  improve  the  system  performance),  emergency  repairs,  and  changes 
resulting  from  regulations. 
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Figure  2  System  Change  Process 
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It  is  the  responsibility  of  the  ASD  to  assign  the  different  kinds 
of  changes  to  specific  releases  of  the  system.  These  assignments  depend 
on  the  amount  of  work  required  by  the  change  and  the  staff  available  to 
work  on  It.  Highest  priority  items  are  given  first  consideration  and  the 
others  are  included  within  the  staffing  constraints.  Some  low  priority 
changes  never  get  Implemented  because  of  the  competition  for  staff 
resources. 

At  any  given  time  work  may  be  done  on  several  change  packages.  At 
the  same  time  the  staff  is  answering  user  questions  and  making  emergency 
repairs  to  the  system. 

4.2  EXISTING  SOURCES  OF  DATA 

The  activities  described  In  the  preceding  section  require  the 
recording,  tabulation  and  presentation  of  data  as  a  necessary  part  of 
their  effective  operation.  The  selection  of  the  data  items,  the 
scheduling  of  their  entry,  reporting,  and  disposal  have  been  designed 
to  support  the  needs  of  the  organizations  that  generate  and  use  them. 
Sometimes  these  needs  are  consistent  with  the  use  of  the  data  for 
research  purposes  and  other  times  they  may  not  be  consistent.  This 
section  describes  the  existing  data  collection  activities;  the  next 
section  consfders  their  usefulness  for  AIRMICS  research  programs. 

Six  sources  of  data  have  been  Identified  as  having  potential  for 
providing  research  data.  They  are: 

•  Resource  Management  Accounting,  Reporting  and  Control 
System  (REMARCS) 

t  Project  Management  System  (PMS) 

•  Incident  Reporting  System 

•  SCR  Status  Accounting  and  Reporting  System 
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These  potential  sources  of  data  were  identified  during  two  series 
of  interviews  with  USACSC  personnel.  The  descriptions  that  follow  are 
based  on  these  discussions  and  pertinent  documents  and  regulations. 

4.2.1  Resource  Management  Accounting,  Reporting  and  Control  System 

( REMARCS  1  [8] 

REMARCS  is  the  primary  means  for  recording  and  reporting  the 
expenditure  of  personnel  resources  at  the  USACSC.  All  organizations  are 
required  to  use  it.  It  is  primarily  used  to  account  for  the  utilization 
of  personnel  in  both  directly  applied  activities  and  supporting  activities. 
It  includes  administrative  resources.  Primary  reporting  Is  at  the  divi¬ 
sion  level  and  above. 

The  functional  proponent  for  REMARCS  is  the  Office  of  the 
Comptroller,  USACSC.  The  Assigned  System  Developer  is  the  Command 
Information  Services  Division  (CISD)  of  the  Intelligence,  Plans  and 
Operations  Office. 

Figure  3  shows  the  form  used  to  record  data  in  REMARCS.  The 
data  items  that  have  possible  significance  for  research  purposes  are: 


Number  of 

Data  Item  Characters 

Organization  Code  4 

Job  Number  6 

Work  Measurement  Code  4 

Skill  Code  1 

Ordinal  Date  5 

Hours  6 

Type  Hours  1 

DPI  Code  4 

SCR  Sequence  Number  3 

Employee  ID  Number  4 
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These  items  identify  hours  of  effort  expended  by  an  individual 
and  describe  the  system  being  worked  on,  the  life  cycle  phase  and  the  type 
of  effort  (e.g.,  analysis,  programming,  testing).  The  data  are  reported 
semi-monthly  and  according  to  Regulation  37-9,  2-1,  all  USACSC  employees 
are  to  report  their  hours  in  REMARCS.  Civilian  hours  must  be  reconcilable 
with  the  hours  reported  for  pay  purposes  and  military  hours  must  be 
reconcilable  with  their  normal  work  week.  REMARCS  provides  for  allocating 
an  Individual's  hours  among  different  activities.  Administrative  and 
supervisory  personnel  are  included  in  the  reporting. 

Data  items  of  particular  interest  are  those  that  serve  to  describe 
the  amount  of  effort,  the  type  of  effort,  who  did  it,  and  what  it  was 
applied  to.  Together  these  describe  the  personnel  resources  applied  to 
produce  the  various  computer  system  components.  This  information  is 
required  in  many  research  programs. 

4. 2. 1.1  Description  of  Effort 

The  Work  Measurement  Code  (WMC)  and  the  Skill  Code  describe  the 
work  being  done. 

The  WMC  indicates  the  particular  part  of  the  system  life  cycle  in 
which  the  effort  was  expended.  The  cycle  is  divided  into: 

•  Project  Phase 

•  Major  Milestone 

•  Detail  Milestone 

•  Task. 
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Appendix  IV  contains  a  list  of  WMCs  taken  from  Reg.  37-9. 

The  major  life  cycle  phases  are: 

•  Planning  and  Definition 

•  Development 

•  Installation,  Operation,  and  Maintenance. 

Within  the  System  Development  Phase,  the  Major  Milestones  include: 
e  Systems  Software  Design 

•  RFP  for  ADPE  Issued 

•  Systems  Programming  and  Documentation 

•  Etc. 

Detailed  Milestones  in  the  Systems  Software  Design  Milestone 
include: 

t  Detailed  System  Design 

•  System  Test  Plans 

•  ADP  Structure  Charts 

•  Etc. 

Tasks  are  presently  used  to  describe  Maintenance  Phase  Activities. 
Under  the  Major  Phase,  Maintenance,  and  the  sub-category,  Emergency/Urgent 
System  Change  Requests,  are  entered  the  Tasks: 

•  Review  and  Analysis 

•  System  Design 
a  Programming 

•  Testing 

9  Documentation. 
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There  is  a  set  of  WMCs  that  describe  Administrative/Management 
and  General  Overhead  activities. 

The  lower  level  code  designations  are  used  in  conjunction  with 
specific  categories  and  sub-categories. 

The  Skill  Code  is  used  along  with  the  WMC  to  provide  information 
indicating  the  relationship  of  the  effort  to  the  primary  function  of 
developing  and  maintaining  computer  systems.  Figure  4  describes  how 
the  Skill  Code  is  used  to  categorize  effort  by  functional  category  and 
skill  type.  Appendix  V  is  a  complete  description  of  Skill  Codes  taken 
from  Reg.  37-9. 

The  Type  Hours  entry  can  be  used  to  identify  hours  that  were 
accomplished  in  an  overtime,  compensatory  or  gratuitous  classification. 

It  also  identifies  sick  leave,  holiday,  and  annual  leave  hours. 

4. 2. 1.2  Application  of  Effort 

Computer  systems  are  the  product  of  all  USACSC  effort.  The  Work 
Measurement  Code  and  Skill  Code  indicate  the  type  of  effort  that  was 
expended  and  whether  it  was  directly  or  indirectly  associated  with  the 
primary  function.  In  order  to  measure  the  effectiveness  of  the  different 
types  of  effort  it  is  necessary  to  know  the  specific  product  that  resulted 
from  the  combined  effort  of  associated  personnel. 

The  Job  Number,  Data  Processing  Installation  (DPI)  Code  and 
System  Change  Request  (SCR)  Sequence  Number  are  used  in  specified  combi¬ 
nations  to  identify  what  was  worked  on  during  a  given  reporting  period. 

Job  Numbers  are  assigned  by  the  Directorates  and  are  not  required 
to  conform  to  any  particular  format.  Their  purpose  is  to  describe  the 
project,  task  or  function  to  which  the  effort  being  reported  was  applied. 
In  general  the  first  two  positions  of  the  Job  Number  indicate  standing 


Skill  Type  of  Effort 


Major  Functional  Categories 


Codes/Nomenclature 


Direct 

Effort 


A/Systems  Analysis  &  Design 


D1 rect 
Support 


Tech 

Support 


Mission 

Mgmt 


B/ Computer  Programming 


C/Testing 


D/ Other  Direct  Effort 


E/Planning 


F/Other  ADP  Operations 


G/ Documentation 


H/System  Clerical  Personnel 


J/Other  Direct  Support 


K/Mission  Mgt.  ADP  Pers. 


L/Mission  Mgt-  Clerical  Pers. 


M/Mission  Mgt.  Other 


Figure  4.  Summary  of  Skill  Codes  and  Functional  Categories 
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functions  of  an  organization  or  long  term  projects  such  as  STAMMIS.  The 
remaining  positions  usually  indicate  more  detailed  breakdowns  of  the 
efforts.  Figure  5  describes  representative  Job  Numbers  for  a  STAMMIS, 
a  non-standard  software  function  and  a  major  support  function. 

Some  Job  Numbers  call  for  specific  Work  Measurement  Codes. 

Job  Numbers  are  also  used  to  describe  Administrative/Management, 
General  Overhead  functions  in  detail. 

The  DPI  Code  position  designates  an  organizational  activity  that 
has  automatic  data  processing  equipment  or  expends  ADP  resources.  However, 
the  position  is  also  used  to  indicate  work  done  on  an  Emergency/Urgent 
Change  Package  (EUCP)  or  a  System  Change  Package  (SCP). 

If  the  reported  work  is  related  to  a  System  Change  Request  (SCR), 
the  Sequence  Number  issued  by  the  initiating  organization  is  entered. 

4. 2. 1.3  Summary 

The  data  recorded  in  REMARCS  Indicate  the  number  of  hours  a  specific 
individual  worked  on  a  particular  task  attributable  to  some  major  function 
or  standard  computer  system.  The  data  Indicate  whether  the  effort  was 
direct  or  indirect  and  the  type  of  work  performed  (e.g.,  analysis,  coding, 
testing). 

The  reporting  coverage  includes  all  USACSC  personnel  and  all  hours 
worked.  The  system  includes  editing  functions  to  ensure  completeness  of 
reporting. 

4. 2. 1.4  REMARCS  Reports 

REMARCS  produces  many  standard  reports  and  can  produce  others  on 
request.  Figure  6  shows  a  report  of  total  paid  civilian  hours  by  project 
and  skill  categories. 
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10  11  12  13  14  15 

STAMMIS  Job  Number  N  G  A  B  A  A 

Logistics  Systems  Job  Code 
NG  =  SAILS  AB 
A  =  Filler 

B  =  Basic  Supply  Cycle  Subsystem 
AA  *  Filler 

WMC  =  Defined  according  to  Appendix  IV 


Non-Standard 

10  11  12  13  14  15  16  17  IB  19 

Software  Job  Number  ll^AAAAAAA 

Technical  Evaluation  and  Support  Directorate  Job  Code 
13  *  Command  Standards  Program 
VA  «  METAC0B0L 
AA  *  Filler 
WMC  *  Filler 


Major  Support 

10  II  12  13  14  15  16  17  18  19 

Function  Job  Number  Z.DAAAAAAAA 

Support  Operations  Directorate  Job  Code 
7D  »  ADPE  Systems  Reference  Library 
A. . .A  *  Filler 

Figure  5  Representative  Job  Numbers 
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Figure  6.  REMARCS  Report  of  Hours  by  Project  and  Skill  Categories 


4.2.2  The  Project  Management  System  (PMS) 

REMARCS  is  designed  to  be  a  system  for  administrative  planning  and 
budgeting  on  an  organizational  level.  Its  purpose  is  to  provide  top 
management  with  information  on  staffing  as  related  to  missions  as  a  follow 
up  to  the  budgeting  process.  The  Project  Management  System  (PMS)  uses 
similar  data  items  for  a  different  purpose.  It  is  designed  to  meet  the 
needs  of  the  project  manager.  It,  therefore,  is  task  rather  than  organi¬ 
zation  oriented  and  is  more  directed  to  how  direct  effort  is  applied  to 
specific  activities  than  in  reconciling  the  efforts  of  all  the  personnel 
in  an  organizational  element. 

PMS  also  describes  how  the  different  tasks  relate  to  one  another 
so  that  the  effects  of  changes  in  the  completion  times  of  the  different 
tasks  can  be  automatically  linked  to  changes  in  completion  times  for 
succeeding  activities  Including  the  completion  date  of  the  project. 

PMS  makes  it  easy  for  a  project  manager  to  study  alternate  staffing 
plans  for  a  given  workload.  It  also  helps  to  schedule  the  completion  of 
a  project  given  a  fixed  amount  of  available  staff  time. 

After  a  project  plan  has  been  implemented,  PMS  provides  periodic 
reports  that  tell  a  manager  how  well  the  schedule  is  being  met.  It  then 
supports  changing  the  schedule  to  accomodate  the  actual  experience. 

4.2.3  The  Incident  Reporting  System 

Whenever  a  Data  Processing  Installation  (DPI)  contacts  the  Customer 
Assistance  Cffice  (CAO)  with  a  problem  whether  the  problem  is  the  proper 
interpretation  of  a  data  item  or  failure  of  a  system  to  execute,  the 
member  of  the  CAO  servicing  the  contact  completes  an  Incident  Report 
(USACSC  Form  53,  Figure  7).  Sometimes  the  problem  Is  corrected  over 
the  telephone  and,  if  this  is  the  case,  an  entry  is  made  on  the  report 
indicating  the  manner  in  which  the  incident  was  closed.  Other  problems 
may  not  be  so  easy  to  solve;  they  may  require  modifications  to  the  system 
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Figure  7.  Incident  Report  (USACSC  Form  53) 


ORIGINATOR  NUMSSR 


INOOSNT  REPORT 


L  CA0  0t90Sm0N 

A.  Q  CAO  RESOLUTION  PROVIDED  AS  WOWN  IN  SLOOU  »  THROUOH  11  SELOW. 

S.  r*1  INCIDENT  CANNOT  IS  RESOLVED  IT  CAO.  RORWAROSD  TO  ASS  ROR  ACTION  A  NO 

LJ  RESOLUTION  ON  _ _  .  VTA  (“1  RHONE  f“ [TWX  plTSUCORT 

(OATB/TIMSI  l— l  l—l 

Qmail  Qcourisr 
□  other  _____________ 

1  METHOD  OR  RESOLUTION 

A.  Q  USSR  REQUESTED  CANCELLATION  (DESCRIES  IASU  ROR  CANCELLATION  IN  COMMENTS  SSLOWI 

a.  Q  USSR  PROBLEM  IS  A  DUPLICATE  OR  IR/SCR  _  OATXD  _ _ 

&  Q  CUSTOM  SR  ASSIST  AN  Cl  RURNISMEO  lOCSCAllS  ASSISTANCE  IN  COMMENTS  BELOW) 

0.  Q  CONVSRTIO  TO  EMERGENCY  SCR  _  PROJECTED  ROR 

INCLUSION  IN  EUCR  RELEASE  DATE  0/A  ____________ 

t  □  CONVERTED  TO  URGENT  SCR  _____________  PROJECTED  ROR 

INCLUSION  IN  SCR  RELEASE  OATE  O/A  ____________ 

R.  Q  USER  IS  REQUESTED  TO  SUBMIT  A  ROUTINE  SCR  (AW  AR  1S-1 

G.  Q  COMMENTS.  .. 


ia  REVIEW 

A.  IN  RESOLVING  THIS  INOOENT  CLOSURE  WAS  OIJCUSSED  WITH  _ 

(NAME) 

OR  _________  ON  _________  AT  ________  HOURS. 

IMIS0/OR1I  (0ATE1 

l  ntM  OR  TASLS  S-1  (ARR  I.  CSCR  1BCT-U  JEST  CATEGORIES  THE  SASJC 

CAUSE  OR  THIS  INOOSNT.  THE  SREORlC  CAUSE  WAS  _ 


_  (ANALYST) 

11.  CLOSURE 

CAO  LOG  CLOSED  _  ET  CONRtRMtNG  MESSAGE 


Figure  7.  (Con't) 


4-24 


pn 


to  correct.  This  determination  is  made  by  the  Assigned  System  Developer 
and  depending  on  the  urgency  of  the  problem  either  an  emergency  repair  is 
scheduled  or  the  change  is  scheduled  for  a  future  release  of  the  system. 
The  basis  for  closing  the  incident  is  always  recorded  on  the  Incident 
Report. 


There  are  two  primary  purposes  of  the  Incident  Report.  One  is  to 
record  problems  experienced  with  a  system  in  a  consistent  manner  so  that 
when  more  than  one  installation  contacts  the  CAO  with  the  same  problem, 
there  will  be  a  means  of  knowing  that  the  second  problem  is  a  duplicate. 
The  second  purpose  of  the  Incident  Report  is  to  ensure  that  a  proper 
response  is  made  by  the  USACSC  to  all  problems  reported  in  the  field. 

Of  course  another  reason  for  keeping  incident  statistics  is  to 
obtain  a  measure  of  the  reliability  of  operational  software.  Poor 
reliability  performance  may  require  a  redesign  or  other  changes  to  a 
system.  Problems  on  a  wider  scale  would  indicate  the  need  for  more  effec 
tive  testing  prior  to  system  release.  In  any  case  a  reliable,  objective 
measure  of  the  error  experience  is  needed. 

The  Incident  Reporting  System  accomplishes  the  following  functions 

•  Records  and  classifies  incidents 

•  Identifies  duplicate  problems 

•  Refers  problems  not  solvable  by  the  CAO  to  the  ASD 

•  Ensures  timely  response  by  the  ASD 

•  Records  the  disposition  of  the  problem 

•  Tabulates  statistics  describing  the  incident  experience  for 
all  systems  (Figure  8). 

These  functions  are  the  responsibility  of  the  Customer  Assistance 

Office. 
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Figure  8  Incident  Statistics 


4.2.4  System  Change  Request  Status  Accounting  and  Reporting  System  [9] 
The  SCR  Status  Accounting  and  Reporting  System  (Proponent  Agency, 
Technical  Evaluation  and  Standards  Directorate  (ACSC-TE);  Assigned  System 
Qeveloper,  Financial  Systems  Directorate  (ASCS-FS))  provides  a  means  for 
entering  new  SCRs  into  the  approval  process  and  to  indicate  the  status  of 
all  SCRs  in  the  data  base.  Daily  updates  are  made  to  the  data  base  from 
sources  that  include  SCR  letters,  DFs,  electrical  messages,  annotated 
SCR  listings,  etc.  .  SCR  status  may  be  queried  by  the  user  agencies 
using  online  terminals.  A  proprietary  data  base  management  system  has 
been  installed  to  provide  this  service.  In  addition,  periodic  status 

reports  are  issued. 

Figure  9  describes  the  SCR  document  (DA  Form  4157-R). 

Figure  10  describes  the  status  designations. 

Figure  11  is  one  of  the  available  summary  reports. 
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Figure  9  SCR  Documentation 
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SCR 

STATUS 

CODE  DESCRIPTION 

A  SCR  Is  not  approved  for  work.  Initial  Impact  using  approved 
resource  estimating  procedures  has  been  requested  by  approving 
authority. 

B  SCR  Is  awaiting  approval  by  the  proponent  agent. 

C  SCR  is  approved  for  work  and  Is  being  impacted. 

D  SCR  reopened  after  being  closed  in  error. 

E  Functional  SCR  approved,  but  additional  information  is  needed 

by  proponent  agent. 

F  SCR  approved  and  awaiting  information  from  the  originator. 

G  Functional  SCR  requiring  three  or  more  manyears  to  accomplish. 

H  SCR  approved  and  placed  in  a  deferred  status. 

J  Technical  SCR  completed  initial  impact  and  awaiting  ARA  approval. 

K  Technical  SCR  approved  and  placed  In  a  deferred  status  by  ARA. 

M  Detailed  impact  completed  and  awaiting  SCR  review  meeting. 

N  SCR  approved  and  being  worked  on. 

Figure  10.  SCR  Status  Descriptors 
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SCR 

STATUS 

CODE 


DESCRIPTION 


P  SCR  is  assigned  to  an  SCP  and  ready  for  or  undergoing  Field 
Validation  Testing. 

R  An  interim  resolution  has  been  broadcast. 

S  SCP  completed  and  Installed. 

T  SCR  resolution  completed  without  change  package. 

W  SCR  disapproved. 

X  SCR  cancelled  by  originator. 

The  following  codes  are  internal  status  descriptors  that  are 
used  in  conjunction  with  Code  ' N ' . 

1  Analysis  and  design. 

2  Programs  are  being  revised. 

3  Pro grams/ system  revised  and  being  tested. 

4  System  documentation  is  being  revised/ developed. 

5  Resolution  to  SCR  is  being  validated.  DCT  level  III. 

6  Documentation  at  QAD  for  certi flcatlon. 

Figure  10  (cont. ) 
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SCR 

STATUS 

CODE  DESCRIPTION 

7  SCR  in  change  package  undergoing  environment  test. 

8  SCR  in  change  package  being  prepared  for  broadcast. 


Figure  10  (cont. ) 
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ANALYSIS  OF  DATA  SOURCES 

The  preceding  sections  have  described  the  need  for  data  to 
support  the  AIRMICS  research  programs  and  the  data  that  are  presently 
being  recorded  at  USACSC  as  a  part  of  normal  operations.  This  section 
analyzes  the  possibilities  for  using  these  existing  data  sources  to 
satisfy  the  research  needs.  In  keeping  with  the  three  general  classi¬ 
fications  of  software  data,  the  analysis  will  evaluate  the  data  according 
to  its  usefulness  in  describing  the  following  attributes  of  software 
systems: 

a  Product 
t  Resources 
•  Environment 

5.1  LEVEL  OF  DETAIL 

Our  objective  in  obtaining  data  is  to  relate  the  application  of 
resources  to  the  quality  of  the  final  product.  The  purpose  of  doing  this 
is  to  learn  how  to  better  use  resources  to  produce  better  products.  It 
is  necessary  as  a  first  step  to  establish  a  common  level  of  detail  at 
which  to  describe  the  resource  and  product  elements. 

We  could  collect  information  at  the  system  ’ -vel .  This  would 
undoubtably  be  the  easiest,  cheapest,  and  most  relic; . -  level  at  which 
to  collect  and  analyze  data.  However,  systems,  especially  large  systems, 
are  not  homogeneous.  Therefore,  it  would  be  unlikely  that  we  could 
identify  similarities  that  would  enable  us  to  draw  inferences  about  the 
relationship  between  the  application  of  resources  and  the  resulting 
product. 

We  must  attempt  to  isolate  homogeneous  elements  in  order  to  make 
valid  comparisons.  The  smaller  the  element  the  fewer  characteristics  it 
has  and  the  easier  it  is  to  describe.  We  are  also  more  likely  to  find 
others  like  it  so  we  can  stratify  our  data.  We  can  therefore  control 
some  variables  so  that  we  can  more  easily  find  relationships  among  the 
others . 


The  lowest  level  of  detail  might  be  the  function.  These  are  the 
elements  of  any  process.  Especially  In  a  given  environment  they  tend 
to  be  the  things  we  do  over  and  over  again.  However,  although  It  may  be 
possible  to  Isolate  functions  In  a  delivered  product,  It  Is  not  feasible 
to  identify  resources  at  that  level  without  systems  that  are  considerably 
more  modular  than  Is  believed  presently  exist. 

Perhaps  a  reasonable  compromise  is  the  program  level.  Although 
not  primitive  or  elementary  functions,  programs  tend  to  be  or  should  be 
functional  at  a  higher  level  of  abstraction  than  the  module. 

We  will  consider,  then,  how  to  describe  three  attributes  of  the 
software  domain  (product,  resources,  and  environment)  at  the  common 
level  of  detail  of  the  program. 

5.2  DATA  DESCRIBING  THE  SOFTWARE  PRODUCT 

The  software  product  Is  considered  to  be  all  the  programs  and 
documentation  that  result  from  the  application  of  resources  in  the  de¬ 
velopment  and  operating  environments. 

At  least  for  Initial  data  collection  It  is  recommended  that  only 
the  delivered  product  be  considered.  The  delivered  product  is  the 
collection  of  program  versions  that  constitute  a  system  version  existent 
in  an  operating  environment  subsequent  to  acceptance. 

The  stipulation  of  versions  is  important.  It  is  necessary  to 
ensure  that  the  resource  and  environment  variables  apply  to  the  proper 
descriptions  of  the  product  they  effected. 

The  use  of  the  delivered  product  limits  studies  to  the  effects 
of  the  integrated  development  and  maintenance  process  on  a  final  product. 
But,  the  alternative  Is  to  try  to  capture  the  dynamics  of  the  development 
process  which  establishes  more  stringent  requirements  for  data  recording. 
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We  believe  that  it  would  be  unwise  to  attempt  this  until  we  have 
successfully  implemented  procedures  to  capture  the  static  data.  The 
history  of  attempts  to  do  this  indicates  that  it  is  a  formidable  under¬ 
taking  by  itself. 

By  limiting  the  product  measurement  to  the  delivered  item,  it  is 
possible  to  apply  any  number  of  measures  without  the  problem  of  data 
perishability.  Once  a  copy  of  the  system  has  been  obtained,  it  can  be 
examined  indefinitely  using  present  and  future  metrics.  A  number  of 
metrics  are  proposed  for  describing  program  attributes  and  several  of 
them  are  automated.  These  will  become  more  standardized  and  more 
automated  in  the  future. 

5.2.1  Description  of  System  Physical  Attributes 

In  addition  to  its  obvious  descriptors  such  as  size,  language, 
and  type  application,  the  product  definition  must  include  a  large  number 
of  metrics  that  completely  specify  the  attributes  of  the  product. 

Table  2  is  a  list  of  product  metrics  compiled  from  several  sources. 
Notice  that  reliability  is  a  product  attribute  and  that  its  description 
i  'ludes  the  classification  of  errors  found  in  the  product.  Errors  can 
occur  in  either  the  software  or  the  documentation. 

Error  classification  schemes  have  been  proposed  by  several 
researchers.  The  earlier  attempts  were  ad  hoc  and  not  well  strsi tured 
for  software  research.  ^  However  some  of  the  later  classification 
schemes  show  promise  of  describing  errors  in  a  way  that  should  permit 
the  investigation  of  how  and  when  errors  are  introduced  into  a  system, 
how  they  might  best  te  detected,  and  how  they  can  be  avoided.  Error 
data  and  its  classification  is  discussed  in  detail  in  Section  5.2.3. 
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TABLE  2 


PRODUCT  METRICS 


SIZE 


NUMBER  OF  LINES  OF  SOURCE  CODE  BY  PROGRAM 
NUMBER  OF  NEW  LINES  OF  SOURCE  CODE  BY  PROGRAM 
NUMBER  OF  STATEMENTS  BY  PROGRAM 
NUMBER  OF  EXECUTABLE  STATEMENTS  BY  PROGRAM 
NUMBER  OF  COMMENT  LINES  BY  PROGRAM 
WORDS  OF  STORAGE  FOR  EACH  NEW  OBJECT  PROGRAM 
WORDS  OF  STORAGE  FOR  NEW  OBJECT  PROGRAM  EXECUTABLE  CODE 
WORDS  OF  STORAGE  FOR  LINKED  CODE 
NUMBER  OF  SUBSYSTEMS 
NUMBER  OF  PROGRAMS 
NUMBER  OF  MODULES 
NUMBER  OF  FUNCTIONS  BY  PROGRAM 
NUMBER  OF  INPUT  FILE  FORMATS  BY  PROGRAM 
NUMBER  OF  OUTPUT  FILE  FORMATS  BY  PROGRAM 

LANGUAGE 

QUALITY  CHARACTERISTICS* 

[11]  UNDE  RSTAN  DAB I L I TY 
COMPLETENESS 
CONCISENESS 
PORTABILITY 
CONSISTENCY 
MAINTAINABILITY 
TESTABILITY 
USABILITY 
RELIABILITY 
STRUCTUREDNESS 
EFFICIENCY 

RELIABILITY  CHARACTERISTICS 

NUMBER  OF  INCIDENT  REPORTS  BY  SYSTEM,  SYSTEM  CHANGE  PACKAGE,  TIME, 
AND  SEVERITY. 

LINES  OF  CODE  CHANGED  BY  SYSTEM  AND  SYSTEM  CHANGE  PACKAGE. 

NUMBER  OF  ERRORS  BY  PHASE  INTRODUCED,  CAUSE,  SEVERITY. 


[12]  RELIABILITY 
FLEXIBILITY 
STRUCTURE 
PERFORMANCE 


*  The  terms  are  defined  in  the  cited  references. 
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TABLE  2  (Cont.) 


DOCUMENTATION  CHARACTERISTICS 

NUMBER  OF  DOCUMENTS 
EXTERNAL 
INTERNAL 

NEW  PAGES  WRITTEN  BY  DOCUMENT  TYPE  AND  LEVEL  OF  SPECIFICATION. 
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Table  2  includes  some  attributes  such  as  complexity  measures 
that  are  difficult  to  apply  to  large  programs  except  by  automated  means. 
Even  such  common  measures  as  lines  of  source  code  are  difficult  to  obtain 
manually  If  certain  subsets  are  required  such  as  lines  of  executable  code. 
Therefore,  it  seems  that  any  comprehensive  study  of  product  measures 
will  require  the  availability  of  code  scanning  programs  to  apply  the 
different  measures.  These  would  permit  very  complete  descriptions  of 
the  product  attributes  and  could  be  changed  as  required.  Discussions 
with  AIRMICS  staff  have  indicated  that  some  automated  program  scanners 
are  already  available  and  others  are  being  planned. 

Given  the  means  to  extract  product  measures,  the  only  data 
requirement  is  the  source  programs  themselves.  Therefore,  it  is  recom¬ 
mended  that  AIRMICS  be  routinely  given  copies  of  new  and  revised 
programs  as  they  are  released  to  the  field*.  It  would  then  be  possible 
to  apply  whatever  metrics  are  needed  for  any  given  research  objective. 

It  would  also  be  possible  to  relate  the  program  code  attributes  and  the 
resources  that  created  the  code. 

Descriptions  of  the  documentation  can  be  obtained  from  the 
creating  organization  without  too  much  difficulty. 

Existing  automated  product  analyzers  measure:^11^ 

•  Size 

Lines  of  code 

Numbers  of  statements  by  type 
Words  of  executable  code** 

Statements  changed 

*  A  standard  2400  reel  of  tape  at  a  1600  BPI  density  can  hold  a  300,000 
line  program. 

**  This  measure  requires  either  the  availability  of  link  editors  and 

library  software  or  the  inclusion  of  the  measures  obtained  when  creatinq 
the  executable  module  for  operational  use. 


•  Complexity 

Various  complexity  metrics 

•  Structuredness 

•  Testing  Completeness 

•  Standards  Compliance 

Other  product  Metrics  include: 

•  Portability 

•  Reliability 

•  Maintainability 

•  Etc. 

Software  Science  Includes  the  relationships  between  various 
program  characteristics  and  certain  measures.  These  measures  are  derived 
by  counting  operators  (m)  and  operands  (n2)  in  the  program.  From  these 
fundamental  metrics  are  derived  such  Software  Science  metrics  as  Program 
Length,  Level,  and  Volume,  Language  Level  and  others.  These  in  turn  are 
related  to  such  Important  measures  as  coding  effort  and  difficulty.  The 
measurement  of  the  fundamental  Software  Science  variables  for  any  given 
program  language  is  readily  automated  and  AIRMICS  has  contracted  to 
obtain  an  automated  tool  for  measuring  the  variables  for  COBOL  programs. 
Therefore,  the  measurement  of  the  Software  Science  parameters  requires  only 
the  source  programs.  However,  data  to  establish  the  relationships  among 
these  measures  and  other  program  characteristics  and  also  resource 
utilizations  must  be  provided  if  the  data  needs  of  the  Software  Science 
program  are  to  be  satisfied. 

5.2.2  System  Reliability  Data 

Most  of  the  quality  metrics  are  obtained  by  examining  the 
programs.  However,  reliability  must  be  measured  In  the  field  over  a 
period  of  time.  Two  measures  of  reliability  are:  Mean  Time  to  Failure; 
and  Number  of  Reported  Errors.  The  second  measure  may  be  related  to  the 
first  and.  Indeed,  several  researchers  have  developed  such  relationships. 
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Failures  of  the  systems  in  the  field  are  reported  to  the 
Customer  Assistance  Office.  Each  reported  failure  generates  an 
Incident  Report  (USACSC  Form  53,  Figure  7).  The  report  indentifies 
the  circumstances  surrounding  the  failure  and  ascertains  that  a  failure 
has  occurred.  Information  is  recorded  that  identifies  the  system  cycle, 
the  possible  cause  of  the  failure  and  when  it  occurred.  These  data  are 
sufficient  to  measure  the  system  failure  rate. 

The  number  of  reported  errors  is  simply  a  tabulation  of  the  data 
on  the  Incident  Report. 

Of  greater  Importance  than  simply  the  number  of  errors,  is 
information  describing  the  types  of  error  that  occur.  This  information 
is  essential  for  investigating  techniques  of  programming  and  testing 
that  either  prevent  errors  or  detect  them  before  they  are  released  to 
the  field. 

5.2.3  Sources  of  Error  Data 

The  Incident  Report  indicates  how  the  reported  problem  was 
resolved.  If  the  Incident  is  a  valid  problem,  Section  10  of  the  report 
describes  the  particular  error  that  cuased  the  problem.  This  determi¬ 
nation  Is  made  by  the  Assigned  System  Developer  according  to  the  classi¬ 
fication  shown  in  Table  3. 

The  present  descriptions  of  errors  used  with  USACSC  Form  53  are 
not  specific  enough  for  research.  The  existing  categories  give  broad 
indications  of  the  reason  for  the  system  failure.  The  identification 
indicates  whether  proper  procedures  were  followed  by  the  user  (if  the 
procedures  are  explicit  and  the  user  did  not  follow  them,  then  tech¬ 
nically  no  error  exists),  inaccuracies  or  omissions  In  procedures  or 
documentation,  faulty  hardware  or  operating  software  or  a  program  error. 
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TABLE  3 

INCIDENT  REPORT  (IR)  CAUSES 


1.  Volume  II  User  Procedures  Not  Followed 

2.  Volume  II  User  Procedures  Not  Understood 

3.  Volume  II  User  Procedures  Not  Correct 

4.  Volume  III  ADP  Procedures  Not  Followed 

5.  Volume  III  ADP  Procedures  Not  Understood 

6.  Volume  III  ADP  Procedures  Not  Correct 

7.  EUCP  Instructions  Not  Followed 

8.  EUCP  Instructions  Not  Understood 

9.  EUCP  Instructions  Not  Correct 

10.  SCP  Instructions  Not  Followed 

11.  SCP  Instructions  Not  Understood 

12.  SCP  Instructions  Not  Correct 

13.  Local  DPI  Procedures  Incorrect  or  Conflict  w/other  Documents 

14.  Other  Documentation  (Specify  in  Comments) 

15.  Executive  Software  Program  Problem 

16.  Application  Program  Problem 

17.  User  Caused  JCL  Problem 

18.  ASD  Caused  JCL  Problem 

19.  Faulty  DP  Media  (Cards,  Tapes,  etc)  provided  by  USACSC 

20.  Required  Interfacing  System  Is  not  in  Current  Library 

21.  Change  Package  Not  Received 

22.  User  Requests  System  Improvement  (Routine  SCR) 

23.  User  Not  Alerted  to  Prior  Reported  Problem  (Dupe) 

24.  Hardware  Problem 

25.  Cause  Other  Than  Above  (Specify  in  Comments) 
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All  of  the  categories  are  useful  and  they  serve  to  Indicate 
whether  problems  result  from  failure  to  properly  educate  the  user 
or  whether  the  automated  procedure  specifications  are  incorrect.  There 
is  also  a  classification  that  indicates  that  the  program  is  in  error. 
However,  the  single  category  of  "Application  Program  Problem"  is  not 
sufficient  for  AIRMICS  requirements. 

The  proper  classification  of  program  errors  poses  significant 

problems.  Attempts  to  do  this  In  the  past  have  resulted  in  reports 

of  inconsistency  in  interpretation  of  the  classifications,  resistance 

from  the  developing  organizations  and  other  problems.  Therefore,  the 

ideal  error  classification  system  should  allow  objective  perhaps  even 

automated  assignment  of  errors.  However,  the  classification  of  errors 

by  automated  means  may  be  quite  ambitious  given  the  present  state  of  the 

ri  1 1 

art.  In  a  rather  extensive  study  of  the' error  classification  problem, L 
Boehm,  et  al ,  concluded  that  only  a  portion  of  the  assignments  could  be 
made  automatically. 

We  believe  that  the  best  procedure  is  to  obtain  some  basic 
information  from  the  programmer  making  repairs  to  a  system  as  an  aid  in 
interpreting  changes  to  the  program  observed  by  inspecting  the  code. 

The  prograirnier1  s  information  should  describe  what  was  done  to  correct 
a  reported  problem.  He  should  also  report  if  the  opportunity  was  taken 
to  Improve  the  efficiency  of  the  code  at  the  time  the  correction  was  made. 
Other  Important  Information  includes  whether  previously  undetected  errors 
were  observed  and  corrected  when  repairing  the  original  problem.  None  of 
this  information  is  readily  obtainable  by  examining  the  changes  made  to 
the  code. 

Therefore,  we  recommend  that  the  information  In  Table  4  be 
obtained  from  the  programmer  making  the  correction  to  the  program  and. 
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TABLE  4 

DOCUMENTATION  OF  ERROR  CORRECTIONS 


Specifically,  what  error  caused  the  observed  problem. 

Did  it  result  from  wrong  or  Incomplete  specifications  i.e.,  did  the 
programner  do  what  he  wanted  to  do  and  in  so  doing  caused  the  program  to 
do  something  other  than  what  the  user  wanted. 

Was  the  difference  between  what  the  program  did  and  what  was 
wanted  caused  by  the  users  failure  to  be  complete  and  specific  about  what 
he  wanted. 

Did  the  failure  occur  because  the  specifications  and  detailed 
design  were  not  consistent  with  the  requirements  statement. 

Was  a  single  statement  at  fault  or  was  it  necessary  to  rewrite 
several  statements  or  paragraphs. 

Were  errors  other  than  the  one  that  caused  the  observed  problem 
identified  and  corrected  in  the  process  of  correcting  the  reported 
probl em. 


Was  the  opportunity  taken  to  rewrite  the  code  associated  with  the 
error  to  make  it  more  efficient. 


Was  the  functioning  of  the  system  enhanced  in  any  other  way. 
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in  addition  that  periodic  samples  of  programs  be  obtained  to  analyze 
and  classify  the  errors  occurring  In  the  operational  systems.  The 
material  in  Table  4  could  be  included  in  Form  53. 

In  summary.  It  Is  recommended  that  error  data  be  obtained  from  two 
sources.  First,  an  expanded  description  such  as  shown  in  Table  4 
should  be  Incorporated  Into  Section  10  of  the  Incident  Report.  Second, 
programs  should  be  developed  that  scan  programs  that  are  modified  to 
correct  errors  to  learn  how  the  program  was  changed  from  the  baseline 
version.  This  type  of  data  Is  most  useful  in  developing  Mutation 
Analysis. 

5.3  DATA  DESCRIBING  CODE  DEVELOPMENT  RESOURCES 
5.3.1  The  Work  Breakdown  Structure 

The  total  effort  needed  to  produce  any  given  version  of  a  software 
system  is  the  sum  of  the  personnel  hours  required  to  complete  the  initial 
development  and  each  of  the  subsequent  change  packages.  The  work 
breakdown  structure  presented  in  this  section  integrates  both  the  develop¬ 
ment  and  maintenance  cycles  into  a  single  structure.  This  approach  provides 
a  complete  yet  detailed  representation  of  the  resources  required  to  create 
and  maintain  a  software  system. 

A  straightforward  work  breakdown  structure  (WBS)  of  the  invested 
resources  can  be  developed  as  in  Figure  12.  The  hierarchical  definition 
of  work  elements  allows  complete  description  of  effort  to  the  SCR  level 
of  activity.  However,  it  does  not  provide  for  the  identification  of  work 
done  on  individual  programs  which  is  required  by  the  stated  level  of 
detail.  But  is  is  a  complete  description  of  resources  expended  and  may 
be  suitable  for  some  studies. 
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Figure  12  Work  Breakdown  Structure  Showing  Development  SCP  and  SCR  Relationships 


Figure  13  introduces  the  relationship  of  program  elements  to  the 
WBS  and  indicates  the  problem  that  occurs  when  doing  so.  As  can  be 
seen  from  the  figure,  programs  can  be  affected  by  more  than  one  SCR. 
Obviously  In  an  environment  such  as  exists  at  USACSC  with  many  SCRs  being 
processed  at  any  moment,  it  frequently  occurs  that  one  or  more  changes 
will  affect  the  same  program.  When  this  happens,  the  normal  practice  is 
to  Integrate  the  changes  to  the  program  so  that  the  needed  functions  can 
be  accomplished  efficiently.  This  may  call  for  clearly  separable  work 
efforts  but  most  likely  it  will  not.  In  any  case  it  is  not  believed 
feasible  to  attempt  to  separate  the  efforts  by  SCR. 

The  other  case  depicted  in  Figure  12  is  that  of  one  SCR  affecting 
several  programs.  In  this  case  because  the  programs  are  separate  entities 
performing  distinct  functions,  the  effort  Is  conceptually  separate  and 
even  though  one  person  may  implement  the  changes  to  the  different  programs, 
he  should  be  able  to  allocate  the  time  spent  to  the  separate  programs. 

The  compromise  is  presented  in  Figure  14.  The  recommended  WBS  for 
resource  expenditures  assumes  that  it  is  possible  to  allocate  hours  spent 
working  on  a  System  Change  Package  by  program.  If  all  the  effort  spent 
on  a  given  program  is  the  result  of  a  single  SCR,  then  It  is  possible  to 
report  the  effort  to  the  SCR  level. 

If  resource  expenditures  are  recorded  in  this  manner,  it  should 
be  possible  to  analyze  the  effect  on  resource  consumption  of  Implementing 
more  than  one  SCR  in  a  single  program.  Some  analysts  believe  there  may 
be  a  learning  effect  that  occurs  that  invalidates  the  assumptions  of  the 
current  resource  estimating  procedure  which  addresses  SCRs  separately. 
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Figure  14  Work  Breakdown  Structure  Showing  Separable  Program  Effort 


5.3.2  Using  REMARCS  to  Obtain  Resource  Data 

Figure  15  represents  a  suggested  breakdown  of  personnel  resource 
elements  that  Is  consistent  with  the  data  needs  described  In  Section  3. 

It  is  also  compatible  with  the  product  description.  It  is  an  orthogonal 
descriptive  scheme  that  relates  type  of  effort,  life  cycle  phase  and 
work  package.  The  work  package,  itself,  is  orthogonal  in  that  SCP  and 
SCR  represent  an  independent  path  to  the  system  version  level  of 
reporting,  if  this  classification  of  personnel  hours  is  desired. 

The  lowest  level  of  detail  Is  the  program  and  all  personnel 
resources  Invested  In  the  product  at  that  level  of  detail  are  identified. 
An  alternative  path  (see  Figure  14)  is  available  through  the  SCR  and 
the  SCP.  In  either  case,  the  roll  up  to  the  system  level  description  is 
consistent  with  the  discussion  In  the  preceding  section.  Effort  not 
directly  attributable  to  a  program,  such  as  system  level  or  SCP  testing 
is  described  at  that  level. 

The  definitions  of  work  category  and  system  life  cycle  phases, 
as  discussed  In  Section  4.2,  are  quite  suitable  for  AIRMICS  research 
projects.  Of  particular  Importance  to  understanding  the  relationships 
between  effort  and  product  are  the  Direct  Effort  and  Direct  Support  Effort 
categories.  These  are  the  hours  that  are  associated  with  a  particular 
ADP  system.  The  Table  5  structure  maintains  these  definitions  but, 
where  possible,  they  are  extended  to  the  program  level  of  detail  instead 
of  stopping  at  the  system  level.  As  was  discussed  In  the  preceding 
section,  this  extension  Is  straightforward  except  for  the  case  of  SCRs 
that  affect  more  than  one  program. 


5-17 


TABLE  5 


LEVELS  OF  REPORTING  FOR  LIFE  CYCLE  PHASE  AND  TYPE  OF  DIRECT  EFFORT 


I  Systems  Planning  and  Definition 


Reporting  Level 
System  Program  SCP  SCR 

Direct  Effort 

System  Analysis  and  Design 

X 

Direct  Support  Effort 

Planning 

X 

Documentation 

X 

Clerical 

X 

Other 

X 

II  Systems  Development 


Direct  Effort 

System  Analysis  and  Design  X 

Application  Programming 
Testing  X 

Other  X 


X 

X 

X 

X 


Direct  Support  Effort 
Planning 
Other  ADP 
Documentation 
Clerical 

Other  Direct  Support 
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TABLE  5  (Con't) 


III  Systems  Installation,  Operation  and  Maintenance 


Extension 

Direct  Effort 
Testing 
Other 

Direct  Support  Effort 
Planning 
Other  ADP 

Other  Direct  Support 


Reporting  Level 
System  Program  SCP  SCR 


X 

X 

X 

X 

X 


Maintenance 

Direct  Effort 

System  Analysis  and  Design  X 
Applicat  on  Programming 
Testing  X 

Other  X 

Direct  Support  Effort 

Planning  X 

Other  ADP  X 

Documentation  X 

Clerical  X 

Other  Direct  Support  X 


X 

X 

X 

X 


X 

X 

X 

X 


Modification 

Direct  Effort 

System  Analysis  and  Design  X 

Application  Programming 
Testing  X 

Other  X 


X  X  X 

X  x  X 

X  x 

X  x 
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TABLE  5  (Cont'd) 

III  Systems  Installation,  Operation  and  Maintenance  (Continued) 


Reporting 

Level 

System 

Program 

SCP 

SCR 

Direct  Support  Effort 

Planning 

X 

X 

Other  ADP 

X 

X 

X 

X 

Documentation 

X 

X 

X 

X 

Clerical 

X 

X 

X 

X 

Other  Direct  Support 

X 

X 

X 

X 
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The  reporting  scheme  described  in  Table  5  can  be  accomplished 
using  REMARCS  if  the  proper  conventions  are  employed  when  reporting  per¬ 
sonnel  hours.  The  classification  of  the  hours  is  accomplished  using: 

•  Job  Number 

•  Work  Measurement  Code 

•  Skill  Code 

•  DPI  Code 

•  SCR  Sequence  Number 

These  inputs  to  REMARCS  are  described  in  Section  4.2.1. 

5. 3. 2.1  Systems  Planning  and  Definition 

This  life  cycle  phase  is  identified  by  a  "1"  in  position  16 
of  the  REMARCS  input  record  (USACSC  Form  181 -R).  According  to  Table  5 
the  recommended  level  of  reporting  for  this  phase  is  the  system. 

Having  identified  the  first  phase  of  the  life  cycle,  the  desig¬ 
nated  two-character  REMARCS  code  for  the  project  can  be  used  to  identify 
the  system  in  the  Job  Number  field. 

Unfortunately,  the  Job  Number  is  used  differently  by  the  different 
USACSC  organizations.  Some  organizations  use  a  functional  breakdown 
of  work  effort  (e.g.  AIRMICS)  and  others  are  system  oriented  (e.g.  FSD, 
SGL,  PSD).  Still  others  use  a  combination  of  functional  and  system 
reporting  (e.g.  TESD,  SID).  Management  and  overhead  functions  tend  to 
be  reported  on  an  activity  basis. 

Since  our  primary  objective  is  to  obtain  information  about 
systems,  it  may  be  possible  to  minimize  the  disruption  of  current 
reporting  practices  by  agreeing  on  a  standard  method  of  identifying 
system  related  activities  and  not  attempting  to  change  any  of  the  others. 
This  would  simplify  the  recording  of  data,  but  It  may  significantly 
complicate  the  extraction  of  research  data  if  the  non-system  effort  is 
to  be  analyzed. 
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In  order  to  signify  that  system  related  data  are  being  entered,  it 
is  necessary  to  designate  some  field  outside  the  job  number  that  can 
be  used  consistantly  for  this  purpose.  Otherwise,  it  will  be  necessary 
to  infer  this  by  analyzing  the  Job  Number  entry.  One  possible  way  to 
identify  that  system  data  is  being  entered  is  to  use  a  code  in  the 
Account  Processing  Code  Field  (positions  44-47)  another  is  to  enter  a 
character  in  position  26  which  is  presently  part  of  the  Hours  field. 

If  the  use  of  a  special  indicator  code  for  system  reporting  is 
adopted,  it  is  suggested  that  positions  10  and  11  of  the  REMARCS  input 
record  be  used  for  the  appropriate  two-character  system  code. 

The  use  of  the  appropriate  Work  Measurement  Code  (Appendix  IV) 
will  identify  the  type  of  work  being  performed  on  the  system. 

Examination  of  the  Work  Measurement  Codes  indicates  that 
although  they  are  Tabled  as  milestones,  the  usage  suggests 
activities.  Some  of  the  codes,  in  fact,  are  difficult  to 
accept  as  milestones,  for  example:  Service  Analysis,  Techni¬ 
cal  Evaluation,  Computer  Programming,  and  Data  Conversion.  It 
is  recommended  that  the  WMC  titles  be  rewritten  to  clearly  re¬ 
present  the  activities  comprising  the  phases  and  subphases 
of  the  development  and  maintenance  cycles. 

5. 3. 2. 2  System  Development 

A  '2'  in  position  16  of  the  input  record  indicates  hours  spent 
on  the  second  phase  of  the  system  life  cycle.  The  suggested  levels  of 
reporting  for  this  phase  (Table  5)  are  the  system  and  the  program. 

Table  5  indicates  which  levels  are  appropriate  for  the  activities  in 
Phase  II. 


The  hours  reported  are  coded  in  a  manner  similar  to  that  used 
for  Phase  I.  Using  a  system  reporting  designator,  the  appropriate 
system  code  is  entered  into  positions  10  and  11  of  the  Job  Number  field. 
If  the  hours  are  applied  to  a  particular  program,  as  in  the  case  of 
program  detailed  design,  coding  or  debugging,  the  two-digit  program  code 
is  entered  in  positions  12  and  13  of  the  Job  Number.  Otherwise,  these 
positions  are  left  blank.  In  either  case  the  WMC  for  the  activities 
being  reported  (Table  5)  are  entered  in  the  appropriate  field. 
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5. 3. 2. 3  Systems  Installation,  Operation  and  Maintenance 

The  identification  of  Phase  III  activities  is  more  complicated 
than  the  first  two  life  cycle  phases.  This  is  because  the  third  phase 
is  divided  into  four  sub-phases  and  each  one  has  different  activities 
that  are  applicable  to  different  levels  of  system  definition.  The  four 
parts  to  Phase  III  are: 

•  Extension 

•  Maintenance 

•  Modification 

•  Other  Phase  III 

The  definitions  of  tha  activities  in  these  sub-phases  are  covered  in 
USACSC  Reg  37-9.  The  specific  sub-phase  being  reported  is  Indicated 
by  codes  '4'  through  '6'  in  position  16  of  the  REMARCS  input  record. 

The  proper  level  for  reporting  Extension  hours  (position  16 
contains  '3')  Is  the  system.  In  this  case  the  Job  Number  is  entered  as 
it  is  for  Phase  I  activities.  The  Work  Measurment  Code  (Appendix  IV) 
indicates  the  objective  to  which  the  hours  were  applied. 

Maintenance  and  Modification  hours  (position  16  contains  ’4'  or 
'5')  may  be  reported  at  the  system,  program,  SCP  or  SCR  level  depending 
on  the  activity  being  performed.  The  assignment  strategy  Is  similar 
to  that  for  Phase  II.  If  the  work  is  applicable  to  a  given  program, 
(detailed  design,  coding,  debugging)  that  program  is  identified  in 
positions  12  and  13  of  the  Job  Number  field.  Otherwise,  it  is  reported 
at  the  system  level  and  positions  12  and  13  are  left  blank.  For  changes 
made  to  existing  programs,  the  version  number  is  entered  into  positions 
14  and  15. 

The  maintenance  or  modification  effort  is  associated  with  some 
type  of  formal  change  request.  The  particular  type  of  change  directive 
is  indicated  in  position  17  of  the  WMC  field.  These  directives  may 
be  EUSCRs,  SCRs,  or  other  types.  They  may  affect  more  than  one  program. 


If  this  is  the  case,  the  time  must  be  allocated  among  the  affected 
programs.  The  appropriate  SCR  code  is  obtained  from  the  SCR  form  (DA 
Form  4157-R)  and  is  entered  in  positions  27-29. 

The  relationship  between  SCR  and  SCP  will  have  to  be  obtained 
from  external  records.  There  is  no  room  on  the  form  to  record  this 
information.  This  could  be  done  implicitly  by  grouping  the  SCRs  in 
a  summary  report  or  by  entering  tables  relating  SCRs  with  specific  SCPs 
for  each  system. 

Figure  16  Illustrates  the  coding  scheme  for  hours  of  effort 
completed  during  the  several  life  cycle  phases  and  sub-phases. 

5-3.3  Personnel  Experience 

As  was  indicated  in  Section  3.3,  it  is  necessary  that  we  know 
not  only  the  number  of  hours  by  type  of  work,  but  we  must  also  know  the 
experience  doing  the  same  or  similar  work  that  is  reflected  in  the 
reported  hours. 

The  recommended  measures  of  personnel  experience  are  presented  in 
Table  6.  The  link  between  an  individual's  experience  measures  and  his 
reported  hours  in  REMARCS  is  the  Employee  ID  Number  (positions  40-43, 
USACSC  Form  181-R).  Since  this  number  is  unique  only  by  organization, 
it  should  be  appended  by  the  Organization  Code  (positions  5-8).  Also 
care  must  be  taken  to  preserve  continuity  when  an  individual  changes 
organizations  and,  therefore,  ID  Number. 

The  Comptroller's  Office  can  provide  a  list  of  the  experience 
characteristics  by  REMARCS  ID  Number.  Since  the  list  will  contain  only 
the  ID  Number  and  an  experience  summary,  it  should  not  violate  any 
privacy  restrictions.  The  Information  pertaining  to  specific  types  of 
experience  will  probably  be  obtainable  most  easily  by  questionnaire*. 


*The  information  may  be  available  from  personnel  records,  but  it  may  be 
easier  to  get  by  questionnaire. 
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REMARCS  INPUT  FORM 


Figure  16  Illustration  of  Coding  System  Reporting 


TABLE  6 


INDIVIDUAL  EXPERIENCE  SUMMARY 

EMPLOYEE  ID  NUMBER 
ORGANIZATION  ID 
GS/MILITARY  RATING 
YEARS  EXPERIENCE  AOP 

YEARS  EXPERIENCE  CURRENT  JOB  CLASSIFICATION 

YEARS  EXPERIENCE  USACSC 

YEARS  EXPERIENCE  BY  SYSTEM  TYPE 

YEARS  EXPERIENCE  FINANCE 

YEARS  EXPERIENCE  LOGISTICS 

YEARS  EXPERIENCE  PERSONNEL 

YEARS  EXPERIENCE  BY  PARTICULAR  SYSTEM 

YEARS  EXPERIENCE  BY  COMPUTER  SYSTEM 

YEARS  EXPERIENCE  BY  OPERATING  SYSTEM 

YEARS  EXPERIENCE  BY  LANGUAGE 

YEARS  EXPERIENCE  SYSTEM  ANALYSIS  AND  DESIGN 
YEARS  EXPERIENCE  PROGRAM  DESIGN 
YEARS  EXPERIENCE  CODING 
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Once  it  is  obtained  in  the  proper  form.  It  can  be  maintained  using  the 
reqular  REMARCS  Information. 

5.3.4  Computer  Resources 

Data  describing  the  use  of  computer  resources  Is  readily  captured 
by  the  computer  operating  system.  This  Is  presently  done  on  an  organ¬ 
izational  basis  in  order  to  allocate  the  cost  of  operating  the  computer 
facilities. 

Since  the  program  identifier  is  normally  Included  In  the  data 
available  when  a  program  is  executed,  it  Is  possible  to  record  the 
resources  used  by  program.  This  can  be  subsequently  tabulated  by  SCP, 
system  or  any  other  meaningful  total.  By  combining  the  data  from  the 
operating  system  with  other  data,  it  should  be  possible  to  tabulate 
computer  resources  by  the  life  cycle  phase  in  which  they  were  expended. 

5.3.5  Environment  Data 

The  development  environment  does  not  change  very  rapidly.  Even 
when  deliberate  efforts  are  made  to  introduce  new  methods  of  group 
organization,  code  development  techniques  or  other  management  methods, 
it  takes  some  time  for  an  identifiable  change  in  the  environment  to 
emerge.  Furthermore,  the  environment  is  difficult  to  describe  in 
objective  terms. 

For  these  reasons,  measurement  of  environmental  characteristics 
should  be  done  by  direct  observation  on  a  periodic  basis  rather  than 
by  some  other  direct  or  indirect  measure.  For  example,  rather  than 
ask  the  question,  "Does  your  group  use  structured  coding  techniques?" 
is  to  prepare  a  few  simple  coding  problems  and  observe  the  techniques 
used  by  group  members  to  solve  them  [13].  Much  can  be  learned  about  coding 
practices  this  way. 


Similarly,  organization  of  effort,  design  techniques,  management 
reporting,  seating  arrangements,  access  to  computer  equipment,  type  of 
support  software  used,  testing  methods  and  other  measures  of  environ¬ 
ment  should  be  observed  say  once  a  year  for  about  two  weeks  by  an  AIRMICS 
research  team.  The  interval  between  observations  should  be  lengthened 
or  shortened  depending  on  the  stability  of  the  environment.  Once  a  good 
baseline  is  established  by  careful  observation,  it  may  be  sufficient  to 
perform  another  formal  evaluation  when  informal  observation  suggests  an 
update  is  required. 

Table  7  lists  some  recommended  environment  data  items. 
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TABLE  7 


ENVIRONMENT  DATA 

TYPE  HARDWARE 
SOFTWARE  SYSTEM 

ACCESS  DURING  PROGRAMMING  AND  DEBUGGING 

•  ON-LINE,  INTERACTIVE  TERMINAL 

•  BATCH  TURNAROUND 

-  MORE  THAN  ONCE  PER  DAY 

-  ONCE  PER  DAY 

-  LESS  THAN  ONCE  PER  DAY 

ACCESS  DURING  TESTING 

USE  OF  MODERN  PROGRAMMING  PRACTICES 

AVAILABILITY  OF  DESIGN  LANGUAGES  AND  OTHER  AIDES 

AVAILABILITY  OF  DEBUGGING  AIDES 

AVAILABILITY  OF  TESTING  AIDES 
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6  RECOMMENDATIONS 
Product  Data 

•  Physical  Characteristics 

-  Limit  initial  data  collection  to  released  versions  of 
the  systems 

-  Continue  and  expand  present  plans  to  obtain  programs 
to  automatically  measure  program  characteristics 
from  source  code 

-  Implement  procedures  for  obtaining  copies  of  systems 
as  they  are  released 

-  Design  a  questionnaire  for  obtaining  summary  descrip¬ 
tions  of  all  documentation 

-  Periodically  examine  complete  system  documentation 
to  measure  changes  in  quality 


•  Reliability 

-  Adopt  a  taxonomy  for  the  classification  of  errors  in 
operational  systems 

-  Expand  the  USACSC  Form  53  Incident  Report  to  include 
information  describing  what  was  done  to  correct  the 
problem 

-  Periodically  subject  selected  systems  to  a  thorough 
manual  and  automated  analysis  of  errors 

-  Investigate  methods  for  classifying  errors  automati¬ 
cally 

-  Implement  a  method  for  relating  information  from  the 
expanded  Incident  Report  to  error  classes 


Resource  Data 

•  Personnel 

-  Use  REMARCS  as  the  basis  for  obtaining  personnel 
hours 

-  Implement  a  standard  format  for  recording  the  Job 
Number  that  will  permit  identification  of  hours 
worked  by  program  and  SCR 
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-  Implement  an  automated  system  for  retrieving  information 
from  the  REMARCS  data  base 

-  Design  a  system  for  obtaining  personnel  characteristics 
from  existing  records  supplemented  by  a  questionnaire 

•  Computer 

-  Obtain  reports  from  the  computer  accounting  system 
describing  the  computer  resources  used  by  program 

and  life  cycle  phase.  This  Includes  resources  by  SCR. 


Environment  Data 

•  Design  a  survey  instrument  describing  the  USACSC  environ¬ 
ment 

•  Administer  the  instrument  to  selected  groups  on  an  annual 
basis  or  as  frequently  as  dictated  by  the  rate  of  change 
of  the  environment 
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APPENDIX  I 

SUMMARY  OF  CLASSES  OF  ERRORS  DETECTABLE  THROUGH  MUTATION  ANALYSIS* 


Simple  Error.  These  are  errors  that  are  equivalent  to  the  mutants,  i.e. 

"a  simple  syntactic  or  semantic  program  transformation  such  as  changing 
a  particular  instance  of  a  relational  operator  to  one  of  the  remaining 
operators  or  changing  the  target  of  an  unconditional  transfer  to  another 
labled  target." 

Specifically,  the  COBOL  mutations  include: 

•  Move  implied  decimal  point. 

•  Add  or  subtract  one  from  an  OCCURS  definition. 

•  Insert  FILLER  of  length  one  between  two  record  items  or 
change  FILLER  length  by  one 

•  Transpose  adjacent  elementary  items  in  records 

•  Alter  file  references 

•  Switch  PERFORMS  and  GO  TOs 

•  Change  ROUNDED  to  truncation  and  vice-versa 

•  Change  the  sense  of  a  MOVE 

The  following  Mutations  are  also  employed: 

•  Operator  Mutations 

-  Arithmetic  operator  replacement 

-  Relational  operator  replacement 

-  Unary  operator  replacement 

-  Unary  operator  removal 

-  Unary  operator  insertion 

•  Control  Structure  Mutations 

-  Jump  statement  replacement 

-  Do  statement  replacement 

*  Source:  A.  T.  Ac ree,  R.  A.  DeMillo  et  al  ,  Mutation  Analysis,  School  of 
Information  and  Computer  Science,  Georgia  Inst,  of  Technology,  GIT- ICS-79/08, 
Sept  1979. 
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Dead  Statement.  Unexecutable  code  or  code  that  gives  incorrect  results 
regardless  of  the  data  presented. 

Dead  Branch.  A  section  of  code  that  is  never  traversed. 

Data  Flow  Error.  Errors  resulting  from  consecutive  accesses  to  a  variable 
in  which  the  variable  is: 

Undefined  then  referenced 
defined,  then  undefined 
defined  then  redefined 

Domain  Error.  An  input  value  causes  an  incorrect  path  to  be  executed  due 
to  an  error  in  a  control  statement. 

Missing  Path  Error.  This  occurs  when  a  predicate  is  missing  and  as  a  result 
a  branch  fails  to  occur.  The  data  are  subsequently  acted  upon  by  a  function 
different  from  the  one  intended.  Two  types  of  missing  path  errors  are 
defined : 

1.  Specification  MPE:  Two  cases  treated  differently  in  the 
specifications  are  treated  the  same  in  the  program. 

2.  Computational  MPE:  The  algorithm  requires  a  path  within 
a  given  specified  domain  which  is  not  in  the  program. 

Missing  Statement  Error.  A  needed  statement  is  not  in  the  program. 
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APPENDIX  II 

CLASSIFICATION  OF  PROGRAM  ERRORS  IN  A  MANNER  COMPATIBLE  WITH  MUTATION 
ANALYSIS  CLASSIFICATIONS. 

Simple  Error 

•  Implied  decimal  in  wrong  position  in  numeric  field. 

•  OCCURS  statement  wrong  length 

•  Record  item  size  wrong. 

•  Record  item  position  wrong. 

•  File  referenced  using  wrong  name. 

•  Used  a  PERFORM  instead  of  a  GO  TO  or  vice  versa, 
t  Improper  rounding  or  truncation 

9  Wrong  sense  on  MOVE  statement 

•  Incorrect  arithmetic  operator 

•  Incorrect  relational  operator 

•  Incorrect  constant 

•  Transfer  to  wrong  place  in  program  using  either  PERFORM  or  GO  TO 

•  Extent  of  PERFORM  calculation  wrong. 

Dead  Statement 

Statement  gives  wrong  results  for  all  input  data. 

Dead  Branch 

Section  of  code  not  reachable  by  any  combination  of  input  data. 

Data  Flow  Error 

•  Field  or  data  item  was  accessed  prior  to  its  receiving  the 
needed  data. 

•  Field  or  data  item  was  referenced  after  the  needed  data  was 
no  longer  available. 

•  Field  or  data  item  was  accessed  when  the  needed  data  had  been 
replaced  by  something  else. 
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Domain  Error 

Data  Sensitive  branching  error 
Missing  Path  Error 

•  Two  cases  treated  separately  in  the  specifications  are 
treated  the  same  in  the  program, 
t  Logic  needed  to  distinguish  between  computations  is 
missing  from  the  program. 

Miss'ng  Statement  Error 

Needed  statement  not  in  program. 
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APPENDIX  III 


DATA  MEEDS  ASSOCIATED  WITH  SOFTWARE  SCIENCE  AND  OTHER  AIRMICS  PROGRAMS 

FOR  EACH  MODULE: 

Identifier  (program  and  system) 

Initial  Operational  Date 
Change  History 
Date 
Reason 

Net  size  change 

Personnel  hours  charged 

Software  type/(3atch,  Interactive,  Demand) 

Space  Constraint 
Time  Constraint 

Amount  of  Code  Originally  obtained  from  another  source 
Amount  of  library  code 

Modern  Programming  Practice  in  Development 
Development  Aids 

FOP  EACH  SYSTEM: 

Identi fl er 
Application  type 
Number  of  Subsystems 
Status 

Development 
Released  in  past  year 
1-3  years  operational 
more  than  3  years  operational 
Analysis  hours  for  original  development 
Analysis  hours  to  date 
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Design  hours  for  original  development 

Design  hours  to  date 

Programming  hours  original  development 

Programming  hours  to  date 

Testing  hours  original  development 

Testing  hours  to  date 

Proportion  of  original  design  taken  from  another  system 
Proportion  of  original  code  taken  from  another  system 
Group  responsible  for  original  release  testing 
Formal  testing  methods  used. 

Number  of  hours  operated  per  month 
Number  of  transactions  per  month 
Number  of  files 
permanent 
temporary 
Number  of  inputs 
Number  of  outputs 
Number  of  maintenance  personnel 
Number  of  maintenance  hours  to  date. 

Developer  experience 
In  field 

In  Classification 
With  type  application 
With  specific  application 
With  specific  system 
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Mission  Work  Measurement  Codes 
Phase  I:  System  Planning  &  Definition 
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!|  CC  16-19 
Code  1 


IA1A 
1  A2A 
d  A3A 


Milestone 

No. 


i 

*  Milestone 


Description 


I  A. 


! 


Concept  Certification 


I 

t 

1.  >  GFSR  Submission 

2.  ?  ASRA  Submission 

3.  :  GFSR/M1SEA  Approval  &  DFSR  Guidance 


I  B.  t  Design  Certification 


’’  1B1A  ‘ 

1- ‘ 

DFSR  Submission 

•  *1 B2A 

2.9 

MISEA  Updated 

j  1B3A  • 

3.j 

DFSR  Approvai/PMP  Guidance 

;  1B4A  i 

4.  r 

PMP  Submission  to  HQ  DA 

1B5A  s 

PMP  Approval/DF  SR  Revised 

-  1 B6A  - 

6. 1 

ADPE  Acquisition  Decision 

■  1B7A 

7'3 

DFSR  Base  Established  Change 

.★1B8A  < 

Q  J* 

SDMP/SDR 
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CC  16-19  Milestone 
Code  No. 


2A3. 
★2A4, 
★2A5, 
★2A6 
★ 2A7 
★2A8 
★2A9 


|  2D1 

‘  ★2D2A 
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Mission  Work  Measurement  Codes 
Phase  II:  System  Development 


Systems  Software  Design 

Detailed  System  Design 
System  Test  Plans 

User  Function  Procedure  Development  Initiated 

Service  Analysis 

Data  Flow  Charge 

ADP  Structure  Charts 

Methoiogy  Development 

Data  Base  Design 

Analysis  Review 

RFP  for  ADPE  Issued 

ADPE  Documentation  Requirement  Draft 
ADPE  Specifications 

ADPE  Contract  Award 

Vendor  Proposals  Received 
Technical  Evaluation 
Cost  Evaluation/Negotiation 
ADPE  Selection 

Systems  Programing  and  Documentation 
Computer  Programing 

Programs  Documentation  INCLD  Module  Specification  Development 
Detailed  Test  Procedures 

Software  System  Support  Center  Facilities 

HQDA  Readiness  Review 

Software  Systems  Support  Center  ADPE  installed 

ADPE  Acceptance  Test 
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Mission  Work  Measurement  Codes 
k  Phase  II:  System  Development 


5 

i  CC  16-19  li  Milestone  j 


:gia 

:g2a 


2H1A 

2h:a 

2H3A 
2H4A 
2H5  A 
2H6A 
2H7A 


« 

w 


Code 

>1  No. 

► 

Milestone  Description 

?  II  F. 

? 

w 

•i 

i 

Systems  Integration  Test 

2F1 A 

I-! 

System  Integration  Test  Conducted 

2F2A 

•  1 

2.  i 

SIT  Results/Recommended  Changes  Documented 

2F3A 

4 * 

3.  1 

User  Functional  Procedures 

2F4A 

> 

4.  ; 

Systems  Development  Package  Prepared  &  Forwarded  to  HQDA 

4 

i  n  g. 

s 


l  II  H. 


H 


i. 


5 


1 

*•5 

O-  : 


HQDA  Review/Approval  of  System  Design 

System  Design  Approval 
Prototype  Te;t  Guidance  Provided 

Prototype  System  Installed 

SDP  Revised 

Prototype  ADPE  Installed 

Prototype  Acceptance  Test  (includes  1DCT) 

User  Documentation  Verified 

Data  Conversion 

Instructor  Training 

Prototype  Operational  (added  for  3S  upgrade  I 


in  J- 


S  Prototype  Test  Evaluation  Test  Completed 


2J1A 

{ 

¥ 

1 

1. 1 

Bench  Mark  Rerun 

2J2A 

1 

. .  * 

On  Site  Evaluation 

2J3A 

] 

Evaluation  Report  Submitted  to  HQDA 

2J4A 

5 

I 

HQDA  Evaluation  Approval  Extension  Plan 

e 

t  n  L. 

f 

4 

i 

2L1 A 

j 

1. 1 

Communication  Analyses 

2L2A 

i 

-»  i 

—  i 

Evaluation  of  Communication  .Alternatives 

2L3A 

i 

j 

1 

J 

} 

i 

Communication  Plan 

IV-3 
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System  Extension 

Systems  Development  Package  Provided 

Facilities  Available 

Training  Accomplished 

ADPE  Installed  and  Accepted 

Data  Conversion 

Unique  Applications  Conversion 

Data  Base  Conversion 
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Mission  Work  Measurement  Codes  .  Phase  III  (Continued) 


Major  Phase/ 
Category  CC  16 


Sub -Category 
CC  17 


S  CC  18 


4-Maintenance 


★  *  5-Modification 


6-Other  Phase  III 


B  Emergency/Urgent  System?  A  Filler 

Change  Requests  £ 

C  FASA/ASD  Pre-SC  R  Inci-  a 

dent  Resolution  'i 

*1 

P  Emergency/Urgent  Change!) 

Package  % 


B  Routine  Change  Requests  ( 

User  Initiated  j 

i: 

C  Pnonty  or  Routine  Changd- 

Requests  DA  Proponent 
Initiated  i 

f- 

D  ARA-initiated  Routine  ’ 

SCR  ? 


System  Change  Package  jj 
jj 

PA  Assistance  \ 


Project  Direct  Support  i ij 
Not  Otherwise  Identifi-  £ 
able.  (May  include  Mission  | 
Management  Personnel  II 
working  on  a  specific  sys-  ] 
tern  for  prolonged  peri-  ij 
ods.)  jj 

Participation  in  Opera-  j 
tional  Reviews  , 


r.  (To  be  used  with  Code  4  or  5 
\  in  CC  16  except  when  P  or  E  in 

iCC  17 .See  Notes  A  &  B  below.) 

1 .  Review  and  Analysis 
2.  System  Design 
3.  Programing 
g  4.  Testing 

i  5.  Documentation 

• 

’  Note  A.  (To  be  with  Code  P  in 
j  CC  17) 

j  1 .  Review  ar.d  Analysis 
J  2.  Testing 
;;  3.  Other 
'  4.  On-Site  Activity 
} 

\  Note  B.  Filled  with  A  if  Code 
5  SE  in  CC  1 6  and  1 7  or  6  in 
I  CC  16  is  used. 


T  On-the-Job  Training 
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Administrative/Management 
General  Overhead 


WMC 
CC  16-19 


U00 

1110 

1120 

1130 

1140 

1150 

1160 

1170 


1210 
1220 
1230 
1240 
1250 
12 
12! 

2110 

2120 

2210 

2220 

2310 


Description 

General  and  Miscellaneous 

Administrative  •  General  and  Gerical 

Administrative  -  ADP  Management 

Administrative  -  Planning 

Administrative  -  Preparing  Reports/Budgets 

Conference  •  Unrelated  to  Specific  ADP  Systems 

Non-Command  Related  Activity 

Administrative  -  Liaison  Activity 

Log  Sys  Data  Elements  and  Codes  Std  Program 

Life  Cycle  Management 

AMIS  Muter  Plan 
Project  Master  Plan 
Data  System  Inquiries 
Technical  Support 
BASOPS  Hardware  Plan 
Resource  Estimation  Techniques 
Other  -  Life  Cycle  Management 

Training  Received 
ADP  -  Training  Command  Courses 
ADP  -  Training,  Non-Command  Courses 
Non-ADP  Training,  Command  Courses 
Non-ADP  Training,  Non-Command  Courses 

Job  Orientation  (NTE  30  days)  for  newly  assigned  programmers  and  ADP  Systems 
analyst  personnel  • 


9999 


All  Leave  -  Civilian  and  Military  (see  para  2-2g(2)) 


27  June  1978 


APPENDIX  V 
SKILL  CODES 
(Card  Column  20) 
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1.  Skill  codes  have  multiple  uses,  therefore,  the  code  determinations  and  explanations  are  by  major  func¬ 
tional  categories. 

2.  Direct  Effort.  Direct  effort  is  reported  only  when  a  mission  job  number  appears  in  card  columns  10-15 
and  one  of  the  following  skill  codes  is  used  in  card  column  20. 

a.  Skill  A  •  Systems  Analysis  and  Design  -  Direct  effort  expended  in  the  following  areas: 

(1)  Performance  of  ADP  systems  analysis  and  design  for  assigned  systems. 

(2)  Development  of  conversion  procedures,  including  system  design  and  documentation. 

(3)  Design  of  conversion  programs  during  system  extension. 

(4)  Development  of  detailed  systems  flow  charts: 

(a)  Skill  code  A  may  be  used  with  any  WMC  with  the  exception  of  6  in  CC  16,  Other  Phase  III  Effort. 

(b)  WMC  beginning  with  3B  in  CC  16  and  17.  Code  A  is  to  be  used  only  for  the  design  of  conversion 
programs  during  system  extension. 

b.  Skill  Code  B  -  Programing  direct  effort  chargeable  to  the  following  areas: 

(1)  Applications  Programing: 

(a)  Coding  of  programs 

(b)  Compilation  and  testing  of  programs 

(c)  Debugging  and  correction  of  programs 

(d)  Development  by  programers  of  standard  documentation  relating  to  the  programing  process,  includ¬ 
ing  draft  training  materials. 

*  (2)  Non-applications  Programing.  Performance  of  any  of  the  above  functions  with  respect  to  system 

unique  executive  software.  Skill  code  B  may  be  used  with  any  WMC  with  the  exception  of  those  beginning 
with  6  in  CC  16.  Further,  skill  code  B  should  not  be  used  when  the  effort  expended  is  as  defined  in  oara 
2a(l)  above. 
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c.  Skill  Code  C  -  Testing  -  Direct  effort  expended  in  the  development  of  test  data  banks  and  conduct¬ 
ing  test  bed  system  tests.  The  use  of  this  code  is  confined  to  analyst  and  programer  participation,  and 
should  not  be  confused  with  ADP  operator  personnel  who  fail  under  other  ADP  operations  in  direct  sup¬ 
port  effort. 

d.  Skill  Code  D  •  Other  Direct  Effort  -  Direct  effort  in  other  actions  requiring  the  use  of  systems  ana¬ 
lysts  and  programers,  but  not  including  those  categories  listed  above.  When  used  with  a  WMC  beginning 
with  3  (extension)  in  CC  16.  this  skill  code  identifies  on-site  extension  effort. 

3.  Direct  Support.  All  personnel,  other  than  systems  analysts  and  programers  including  secretarial,  clerical, 
and  higher-level  supervision  who  are  assigned  to  a  specific  mission  system. 

a.  Skill  Code  E  -  Planning  -  Direct  support  applicable  to  mission  job  and  WMC,  includes  the  effort  of 
all  ADP  professionals  in  the  following  functions: 

( 1 )  Assist  HQDA  staff  and  their  designated  agents  in  the  preparation  and  review  of  General  and  De¬ 
tailed  Functional  System  Requirements  (GFSR'DFSR). 

(2)  Prepare  input  to  Economic  Analysis  (EA). 

(3)  Develop  ADP  Project  Master  Plans  (PMP). 

(4)  Develop  Project  Budgetary  Data  and/or  Program  Change  Requests  (PCR). 

(5)  Develop  impact  statements  for  projected  changes  to  systems. 

(6)  Training  of  system  user  cadre. 

(7)  Supervisory  Project  Management. 

(8)  Hardware  Project  Management. 

(9)  ADP  Technical  Standards  Development. 

b.  Skill  Code  F  -  Other  ADP  Operations  •  Direct  support  representing  all  aspects  of  on-site  extension 
and  field  assistance/incident  resolution  effort  not  employing  those  personnel  performing  systems  analysis  or 
programing.  This  effort  includes  the  following: 

( 1 )  Participation  in  readiness  reviews. 

(2)  Participation  in  evaluations. 

(3)  Conduct  of  technical  inspections. 

(41  Quick-fix  assistance:  installation  of  changes,  and  assistance  to  operational  DPI,  including  around 
the-dock  problem  assistance:  incident  resolution  information  or  direction:  and  performance  evaluation  and 
continuing  systems  implementation  assistance. 
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c.  Skill  Code  G  -  Documentation  •  Direct  support  attributed  to  editing,  rewriting,  graphics,  and  typing 
performed  by  non-ADP  personnel  in  preparation  of  systems  documentation. 


d.  Skill  Code  H  •  Clerical  Support  -  Direct  support  contributed  to  system/project  clerical  support  not 
covered  in  documentation  above. 

e.  Skill  Code  J  -  Other  Direct  Support  -  Direct  support  contributed  by  personnel  assigned  to  support  a 
specific  ADP  system  who  are  neither  ADP  professionals,  clerical,  nor  non-ADP  personnel  involved  in  docu¬ 
mentation.  This  category  includes,  but  is  not  limited  to,  project  officers  who  are  not  ADP  professionals, 
functional  systems  specialists,  technical  writers  or  non -clerical  administrative/supervisory  effort. 

4.  Technical  Support.  All  personnel  assigned  to,  or  otherwise  involved  in,  the  effort  described  in  the  de¬ 
tailed  functional  categories  below. 


a.  ADPE  operations  include  the  operations  of  computer  facilities,  contracting  officer  technical  repre¬ 
sentation  in  the  administration  of  ADPE  contracts,  associated  planning,  and  supervision  and  clerical  effort. 
The  skill  codes  used  are  the  same  as  those  for  direct  or  direct  support  type  of  effort. 

(1)  Programers  and  analysts  assigned  to  provide  on-site  software  problem  resolution  in  support  of 
ADPE  Test  Bed  Operations  will  use  one  of  the  following  previously  described  codes,  depending  on  the 
effort  performed: 


Code  Type  of  Effort 

A  System  Analysis/Design 

B  Programing 

(2)  Computer  operators,  aids,  and  xeypunch  operator  personnel  will  use  Code  F  “Other  ADP  Opera¬ 
tions.” 

(3)  All  other  personnel  in  APDE  operations  will  select  appropriate  codes  E-J  as  described  for  direct 
support. 

b.  Executive  Software.  All  effort  related  to  planning,  developing,  extending,  maintaining,  or  modify¬ 
ing  executive  software  applicable  to  more  than  one  STAMMIS  or  major  Tactical  System  for  the  purpose  of 
minimizing  the  functional  programing  effort  and  maximizing  efficient  and  effective  utilization  of  ADP 
hardware  capability.  This  effort  is  identified  by  the  same  codes  used  for  direct  (Codes  A-D)  and  direct 
Support  (Codes  E-J).  Executive  software  here  specifically  excludes  system  unique  Tactical  executive  soft¬ 
ware,  which  is  considered  direct  effort. 

c.  Quality  Assurance.  All  ^ffort  relating  to  the  USACSC  Quality  Assurance  Program  for  all  Command 
automated  data  systems  life  cycle  activities,  to  include  Command  standards,  data  elements  and  code  stan¬ 
dardization.  Normally .  this  effort  will  be  identified  by  use  of  Quality  Assurance  job  codes  with  direct  sup¬ 
port  skill  codes  (Codes  E-J).  However,  when  programing  or  systems  analysis  and  design  effort  is  required, 
the  appropriate  direct  effort  skill  codes  (Codes  A-D)  may  be  used  with  QA  job  numbers. 


V-3 


REG  37-9 
Cl 


22  February  1977 


d.  Advanced  ADP  Technology  effort  is  basically  recorded  using  job  codes  separating  RDTE  and  OMA 
funded  effort.  Advanced  ADP  Technology  effort  will  be  identified  by  direct  support  skill  codes  (Codes  E-J) 
with  appropriate  Advanced  Technology  Directorate  Managed  Job  Codes  (Appendix  D-25).  If  programing  or 
systems  analysis  and  design  are  used  on  Advanced  Technology  tasks,  the  appropriate  direct  effort  codes  will 
be  used  with  these  job  codes. 

e.  ADP  Training  and  Maintenance  of  Technical  Libraries  effort  will  use  the  same  skill  codes  that  are 
used  for  direct  support  (Codes  E-J). 

5.  Mission  Management.  Mission  Management  functions  are  contributed  by  technical  directors  and  their 
secretaries,  a*  -veil  as  direct  support  personnel  who  cannot  be  identified  to  a  single  ADP  system.  Those 
personnel  who  contribute  to  the  support  of  various  projects  but  cannot  be  included  in  any  of  the  foregoing 
functional  categories  will  use  one  of  the  following  skill  codes: 

(1)  Skill  Code  K  -  Mission  Management  ADP  Personnel  -  The  skill  code  K  is  assigned  to  ADP  profes¬ 
sionals  who  are  not  engaged  in  direct  effort  and  whose  duties  are  not  limited  to  a  single  ADP  system. 

(2)  Skill  Code  L  -  Mission  Management  Gerical  •  Assigned  to  clerical  personnel  whose  duties  are  not 
confined  primarily  to  a  single  ADP  system. 

(3)  Skill  Code  M  -  Mission  Management  -  Other  -  Mission  Management  efforts  expended  by  personnel 
such  as  functional  specialists,  contract  specialists,  cost  or  program  analysis,  and  supervisory  personnel  who 
are  not  ADP  professionals,  and  whose  duties  are  not  limited  to  a  specific  system. 

b.  Mission  management  codes  above  are  also  used  with  overhead  job  codes  when  effort  is  concerned 
with  activity  direction,  impacting  only  indirectly  on  specific  projects  as  in  the  assignment  of  personnel  or 
work  on  reorganizations,  etc.  Also  work  performed,  whether  managerial  or  clerical,  spread  over  so  many 
projects  that  accurate  count  cannot  be  kept  of  individual  project  time. 

o  HO  Elements.  This  category  is  limited  to  all  personnel  assigned  to  Headquarter  elements.  HQ  element 
effort  will  use  mission  management  skill  codes  K.  L,  and  M  with  overhead  job  codes.  If  specifically  directed 
to  do  so.  HO  element  effort  may  be  charged  against  special  tasks  which  have  been  assigned  job  numbers  of 
their  own.  using  mission  management  skill  codes. 
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ADP 

ADPE 

AIRMICS 

AMIS 

ASD 

CAO 

CISD 

DFSR 

DPI 

EUCP 

EUCR 

FSD 

GFSR 

HQDA 

IR 

MI  SEA 

PA 

PET 

PMP 

PMS 

PSD 

QAD 

REMARCS 

RFP 

SCP 

SCR 

SOP 

SDR 

SGL 

SID 


APPENDIX  VI 

GLOSSARY  OF  ABBREVIATIONS 

Automatic  Data  Processing 
Automated  Data  Processing  Equipment 
Army  Institute  for  Research  in  Management  Information 
and  Computer  Science 
Army  Management  Information  System 
Assigned  System  Developer 
Customer  Assistance  Office 
Command  Information  Services  Division 
Detailed  Functional  System  Requirement 
Data  Processing  Installation 
Emergency/Urgent  Change  Package 
Emergency/Urgent  Change  Request 
Financial  Systems  Directorate 
General  Functional  System  Requirements 
Headquarters,  Department  of  the  Army 
Incident  Report 

Management  Information  System  Economic  Analysis 

Proponent  Agency 

Prototype  Evaluation  Test 

Project  Master  Plan 

Project  Management  System 

Personnel  Systems  Directorate 

Quality  Assurance  Directorate 

Resource  Management  Accounting  Reporting  and  Control  System 

Request  for  Proposals 

System  Change  Package 

System  Change  Request 

System  Development  Plan 

System  Design  Review 

Support  Group  Lee 

Systems  Integration  Directorate 
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SIT  Systems  Integration  Test 

STAMMIS  Standard  Army  Multi -Command  Management  Information  System 
TCR  Technical  Change  Request 

TESD  Technical  Evaluation  and  Support  Directorate 

USACSC  United  States  Army  Computer  Systems  Command 


